zhangyiss/spack
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

spack containerize generates containers from envs (#14202)

Browse Source 
This PR adds a new command to Spack:
```console
$ spack containerize -h
usage: spack containerize [-h] [--config CONFIG]

creates recipes to build images for different container runtimes

optional arguments:
  -h, --help       show this help message and exit
  --config CONFIG  configuration for the container recipe that will be generated
```
which takes an environment with an additional `container` section:
```yaml
spack:
  specs:
  - gromacs build_type=Release 
  - mpich
  - fftw precision=float
  packages:
    all:
      target: [broadwell]

  container:
    # Select the format of the recipe e.g. docker,
    # singularity or anything else that is currently supported
    format: docker
    
    # Select from a valid list of images
    base:
      image: "ubuntu:18.04"
      spack: prerelease

    # Additional system packages that are needed at runtime
    os_packages:
    - libgomp1
```
and turns it into a `Dockerfile` or a Singularity definition file, for instance:
```Dockerfile
# Build stage with Spack pre-installed and ready to be used
FROM spack/ubuntu-bionic:prerelease as builder

# What we want to install and how we want to install it
# is specified in a manifest file (spack.yaml)
RUN mkdir /opt/spack-environment \
&&  (echo "spack:" \
&&   echo "  specs:" \
&&   echo "  - gromacs build_type=Release" \
&&   echo "  - mpich" \
&&   echo "  - fftw precision=float" \
&&   echo "  packages:" \
&&   echo "    all:" \
&&   echo "      target:" \
&&   echo "      - broadwell" \
&&   echo "  config:" \
&&   echo "    install_tree: /opt/software" \
&&   echo "  concretization: together" \
&&   echo "  view: /opt/view") > /opt/spack-environment/spack.yaml

# Install the software, remove unecessary deps and strip executables
RUN cd /opt/spack-environment && spack install && spack autoremove -y
RUN find -L /opt/view/* -type f -exec readlink -f '{}' \; | \
    xargs file -i | \
    grep 'charset=binary' | \
    grep 'x-executable\|x-archive\|x-sharedlib' | \
    awk -F: '{print $1}' | xargs strip -s


# Modifications to the environment that are necessary to run
RUN cd /opt/spack-environment && \
    spack env activate --sh -d . >> /etc/profile.d/z10_spack_environment.sh

# Bare OS image to run the installed executables
FROM ubuntu:18.04

COPY --from=builder /opt/spack-environment /opt/spack-environment
COPY --from=builder /opt/software /opt/software
COPY --from=builder /opt/view /opt/view
COPY --from=builder /etc/profile.d/z10_spack_environment.sh /etc/profile.d/z10_spack_environment.sh

RUN apt-get -yqq update && apt-get -yqq upgrade                                   \
 && apt-get -yqq install libgomp1 \
 && rm -rf /var/lib/apt/lists/*

ENTRYPOINT ["/bin/bash", "--rcfile", "/etc/profile", "-l"]
```
This commit is contained in:
Massimiliano Culpo
2020-01-31 02:19:55 +01:00
committed by GitHub
parent ed501eaab2
commit 9635ff3d20
24 changed files with 1238 additions and 3 deletions
Download Patch File Download Diff File Expand all files Collapse all files

307
lib/spack/docs/containers.rst Normal file
View File

@@ -0,0 +1,307 @@
.. Copyright 2013-2020 Lawrence Livermore National Security, LLC and other
Spack Project Developers. See the top-level COPYRIGHT file for details.
SPDX-License-Identifier: (Apache-2.0 OR MIT)
.. _containers:
================
Container Images
================
Spack can be an ideal tool to setup images for containers since all the
features discussed in :ref:`environments` can greatly help to manage
the installation of software during the image build process. Nonetheless,
building a production image from scratch still requires a lot of
boilerplate to:
- Get Spack working within the image, possibly running as root
- Minimize the physical size of the software installed
- Properly update the system software in the base image
To facilitate users with these tedious tasks, Spack provides a command
to automatically generate recipes for container images based on
Environments:
.. code-block:: console
$ ls
spack.yaml
$ spack containerize
# Build stage with Spack pre-installed and ready to be used
FROM spack/centos7:latest as builder
# What we want to install and how we want to install it
# is specified in a manifest file (spack.yaml)
RUN mkdir /opt/spack-environment \
&& (echo "spack:" \
&& echo " specs:" \
&& echo " - gromacs+mpi" \
&& echo " - mpich" \
&& echo " concretization: together" \
&& echo " config:" \
&& echo " install_tree: /opt/software" \
&& echo " view: /opt/view") > /opt/spack-environment/spack.yaml
# Install the software, remove unecessary deps
RUN cd /opt/spack-environment && spack install && spack gc -y
# Strip all the binaries
RUN find -L /opt/view/* -type f -exec readlink -f '{}' \; | \
xargs file -i | \
grep 'charset=binary' | \
grep 'x-executable\|x-archive\|x-sharedlib' | \
awk -F: '{print $1}' | xargs strip -s
# Modifications to the environment that are necessary to run
RUN cd /opt/spack-environment && \
spack env activate --sh -d . >> /etc/profile.d/z10_spack_environment.sh
# Bare OS image to run the installed executables
FROM centos:7
COPY --from=builder /opt/spack-environment /opt/spack-environment
COPY --from=builder /opt/software /opt/software
COPY --from=builder /opt/view /opt/view
COPY --from=builder /etc/profile.d/z10_spack_environment.sh /etc/profile.d/z10_spack_environment.sh
RUN yum update -y && yum install -y epel-release && yum update -y \
&& yum install -y libgomp \
&& rm -rf /var/cache/yum && yum clean all
RUN echo 'export PS1="\[$(tput bold)\]\[$(tput setaf 1)\][gromacs]\[$(tput setaf 2)\]\u\[$(tput sgr0)\]:\w $ \[$(tput sgr0)\]"' >> ~/.bashrc
LABEL "app"="gromacs"
LABEL "mpi"="mpich"
ENTRYPOINT ["/bin/bash", "--rcfile", "/etc/profile", "-l"]
The bits that make this automation possible are discussed in details
below. All the images generated in this way will be based on
multi-stage builds with:
- A fat ``build`` stage containing common build tools and Spack itself
- A minimal ``final`` stage containing only the software requested by the user
-----------------
Spack Base Images
-----------------
Docker images with Spack preinstalled and ready to be used are
built on `Docker Hub <https://hub.docker.com/u/spack>`_
at every push to ``develop`` or to a release branch. The OS that
are currently supported are summarized in the table below:
.. _containers-supported-os:
.. list-table:: Supported operating systems
:header-rows: 1
* - Operating System
- Base Image
- Spack Image
* - Ubuntu 16.04
- ``ubuntu:16.04``
- ``spack/ubuntu-xenial``
* - Ubuntu 18.04
- ``ubuntu:16.04``
- ``spack/ubuntu-bionic``
* - CentOS 6
- ``centos:6``
- ``spack/centos6``
* - CentOS 7
- ``centos:7``
- ``spack/centos7``
All the images are tagged with the corresponding release of Spack:
.. image:: dockerhub_spack.png
with the exception of the ``latest`` tag that points to the HEAD
of the ``develop`` branch. These images are available for anyone
to use and take care of all the repetitive tasks that are necessary
to setup Spack within a container. All the container recipes generated
automatically by Spack use them as base images for their ``build`` stage.
-------------------------
Environment Configuration
-------------------------
Any Spack Environment can be used for the automatic generation of container
recipes. Sensible defaults are provided for things like the base image or the
version of Spack used in the image. If a finer tuning is needed it can be
obtained by adding the relevant metadata under the ``container`` attribute
of environments:
.. code-block:: yaml
spack:
specs:
- gromacs+mpi
- mpich
container:
# Select the format of the recipe e.g. docker,
# singularity or anything else that is currently supported
format: docker
# Select from a valid list of images
base:
image: "centos:7"
spack: develop
# Whether or not to strip binaries
strip: true
# Additional system packages that are needed at runtime
os_packages:
- libgomp
# Extra instructions
extra_instructions:
final: |
RUN echo 'export PS1="\[$(tput bold)\]\[$(tput setaf 1)\][gromacs]\[$(tput setaf 2)\]\u\[$(tput sgr0)\]:\w $ \[$(tput sgr0)\]"' >> ~/.bashrc
# Labels for the image
labels:
app: "gromacs"
mpi: "mpich"
The tables below describe the configuration options that are currently supported:
.. list-table:: General configuration options for the ``container`` section of ``spack.yaml``
:header-rows: 1
* - Option Name
- Description
- Allowed Values
- Required
* - ``format``
- The format of the recipe
- ``docker`` or ``singularity``
- Yes
* - ``base:image``
- Base image for ``final`` stage
- See :ref:`containers-supported-os`
- Yes
* - ``base:spack``
- Version of Spack
- Valid tags for ``base:image``
- Yes
* - ``strip``
- Whether to strip binaries
- ``true`` (default) or ``false``
- No
* - ``os_packages``
- System packages to be installed
- Valid packages for the ``final`` OS
- No
* - ``extra_instructions:build``
- Extra instructions (e.g. `RUN`, `COPY`, etc.) at the end of the ``build`` stage
- Anything understood by the current ``format``
- No
* - ``extra_instructions:final``
- Extra instructions (e.g. `RUN`, `COPY`, etc.) at the end of the ``final`` stage
- Anything understood by the current ``format``
- No
* - ``labels``
- Labels to tag the image
- Pairs of key-value strings
- No
.. list-table:: Configuration options specific to Singularity
:header-rows: 1
* - Option Name
- Description
- Allowed Values
- Required
* - ``singularity:runscript``
- Content of ``%runscript``
- Any valid script
- No
* - ``singularity:startscript``
- Content of ``%startscript``
- Any valid script
- No
* - ``singularity:test``
- Content of ``%test``
- Any valid script
- No
* - ``singularity:help``
- Description of the image
- Description string
- No
Once the Environment is properly configured a recipe for a container
image can be printed to standard output by issuing the following
command from the directory where the ``spack.yaml`` resides:
.. code-block:: console
$ spack containerize
The example ``spack.yaml`` above would produce for instance the
following ``Dockerfile``:
.. code-block:: docker
# Build stage with Spack pre-installed and ready to be used
FROM spack/centos7:latest as builder
# What we want to install and how we want to install it
# is specified in a manifest file (spack.yaml)
RUN mkdir /opt/spack-environment \
&& (echo "spack:" \
&& echo " specs:" \
&& echo " - gromacs+mpi" \
&& echo " - mpich" \
&& echo " concretization: together" \
&& echo " config:" \
&& echo " install_tree: /opt/software" \
&& echo " view: /opt/view") > /opt/spack-environment/spack.yaml
# Install the software, remove unecessary deps
RUN cd /opt/spack-environment && spack install && spack gc -y
# Strip all the binaries
RUN find -L /opt/view/* -type f -exec readlink -f '{}' \; | \
xargs file -i | \
grep 'charset=binary' | \
grep 'x-executable\|x-archive\|x-sharedlib' | \
awk -F: '{print $1}' | xargs strip -s
# Modifications to the environment that are necessary to run
RUN cd /opt/spack-environment && \
spack env activate --sh -d . >> /etc/profile.d/z10_spack_environment.sh
# Bare OS image to run the installed executables
FROM centos:7
COPY --from=builder /opt/spack-environment /opt/spack-environment
COPY --from=builder /opt/software /opt/software
COPY --from=builder /opt/view /opt/view
COPY --from=builder /etc/profile.d/z10_spack_environment.sh /etc/profile.d/z10_spack_environment.sh
RUN yum update -y && yum install -y epel-release && yum update -y \
&& yum install -y libgomp \
&& rm -rf /var/cache/yum && yum clean all
RUN echo 'export PS1="\[$(tput bold)\]\[$(tput setaf 1)\][gromacs]\[$(tput setaf 2)\]\u\[$(tput sgr0)\]:\w $ \[$(tput sgr0)\]"' >> ~/.bashrc
LABEL "app"="gromacs"
LABEL "mpi"="mpich"
ENTRYPOINT ["/bin/bash", "--rcfile", "/etc/profile", "-l"]
.. note::
Spack can also produce Singularity definition files to build the image. The
minimum version of Singularity required to build a SIF (Singularity Image Format)
from them is ``3.5.3``.

BIN
lib/spack/docs/dockerhub_spack.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 88 KiB

2
lib/spack/docs/environments.rst
View File

@@ -49,6 +49,8 @@ Spack uses a "manifest and lock" model similar to `Bundler gemfiles
managers. The user input file is named ``spack.yaml`` and the lock
file is named ``spack.lock``
.. _environments-using:
------------------
Using Environments
------------------

1
lib/spack/docs/index.rst
View File

@@ -66,6 +66,7 @@ or refer to the full manual below.
config_yaml
build_settings
environments
containers
mirrors
module_file_support
repositories

25
lib/spack/spack/cmd/containerize.py Normal file
View File

@@ -0,0 +1,25 @@
# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other
# Spack Project Developers. See the top-level COPYRIGHT file for details.
#
# SPDX-License-Identifier: (Apache-2.0 OR MIT)
import os
import os.path
import spack.container
description = ("creates recipes to build images for different"
" container runtimes")
section = "container"
level = "long"
def containerize(parser, args):
config_dir = args.env_dir or os.getcwd()
config_file = os.path.abspath(os.path.join(config_dir, 'spack.yaml'))
if not os.path.exists(config_file):
msg = 'file not found: {0}'
raise ValueError(msg.format(config_file))
config = spack.container.validate(config_file)
recipe = spack.container.recipe(config)
print(recipe)

81
lib/spack/spack/container/__init__.py Normal file
View File

@@ -0,0 +1,81 @@
# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other
# Spack Project Developers. See the top-level COPYRIGHT file for details.
#
# SPDX-License-Identifier: (Apache-2.0 OR MIT)
"""Package that provides functions and classes to
generate container recipes from a Spack environment
"""
import warnings
import spack.environment
import spack.schema.env as env
import spack.util.spack_yaml as syaml
from .writers import recipe
__all__ = ['validate', 'recipe']
def validate(configuration_file):
"""Validate a Spack environment YAML file that is being used to generate a
recipe for a container.
Since a few attributes of the configuration must have specific values for
the container recipe, this function returns a sanitized copy of the
configuration in the input file. If any modification is needed, a warning
will be issued.
Args:
configuration_file (str): path to the Spack environment YAML file
Returns:
A sanitized copy of the configuration stored in the input file
"""
import jsonschema
with open(configuration_file) as f:
config = syaml.load(f)
# Ensure we have a "container" attribute with sensible defaults set
env_dict = spack.environment.config_dict(config)
env_dict.setdefault('container', {
'format': 'docker',
'base': {'image': 'ubuntu:18.04', 'spack': 'develop'}
})
env_dict['container'].setdefault('format', 'docker')
env_dict['container'].setdefault(
'base', {'image': 'ubuntu:18.04', 'spack': 'develop'}
)
# Remove attributes that are not needed / allowed in the
# container recipe
for subsection in ('cdash', 'gitlab_ci', 'modules'):
if subsection in env_dict:
msg = ('the subsection "{0}" in "{1}" is not used when generating'
' container recipes and will be discarded')
warnings.warn(msg.format(subsection, configuration_file))
env_dict.pop(subsection)
# Set the default value of the concretization strategy to "together" and
# warn if the user explicitly set another value
env_dict.setdefault('concretization', 'together')
if env_dict['concretization'] != 'together':
msg = ('the "concretization" attribute of the environment is set '
'to "{0}" [the advised value is instead "together"]')
warnings.warn(msg.format(env_dict['concretization']))
# Check if the install tree was explicitly set to a custom value and warn
# that it will be overridden
environment_config = env_dict.get('config', {})
if environment_config.get('install_tree', None):
msg = ('the "config:install_tree" attribute has been set explicitly '
'and will be overridden in the container image')
warnings.warn(msg)
# Likewise for the view
environment_view = env_dict.get('view', None)
if environment_view:
msg = ('the "view" attribute has been set explicitly '
'and will be overridden in the container image')
warnings.warn(msg)
jsonschema.validate(config, schema=env.schema)
return config

50
lib/spack/spack/container/images.json Normal file
View File

@@ -0,0 +1,50 @@
{
"ubuntu:18.04": {
"update": "apt-get -yqq update && apt-get -yqq upgrade",
"install": "apt-get -yqq install",
"clean": "rm -rf /var/lib/apt/lists/*",
"environment": [],
"build": "spack/ubuntu-bionic",
"build_tags": {
"develop": "latest",
"0.14": "0.14",
"0.14.0": "0.14.0"
}
},
"ubuntu:16.04": {
"update": "apt-get -yqq update && apt-get -yqq upgrade",
"install": "apt-get -yqq install",
"clean": "rm -rf /var/lib/apt/lists/*",
"environment": [],
"build": "spack/ubuntu-xenial",
"build_tags": {
"develop": "latest",
"0.14": "0.14",
"0.14.0": "0.14.0"
}
},
"centos:7": {
"update": "yum update -y && yum install -y epel-release && yum update -y",
"install": "yum install -y",
"clean": "rm -rf /var/cache/yum && yum clean all",
"environment": [],
"build": "spack/centos7",
"build_tags": {
"develop": "latest",
"0.14": "0.14",
"0.14.0": "0.14.0"
}
},
"centos:6": {
"update": "yum update -y && yum install -y epel-release && yum update -y",
"install": "yum install -y",
"clean": "rm -rf /var/cache/yum && yum clean all",
"environment": [],
"build": "spack/centos6",
"build_tags": {
"develop": "latest",
"0.14": "0.14",
"0.14.0": "0.14.0"
}
}
}

72
lib/spack/spack/container/images.py Normal file
View File

@@ -0,0 +1,72 @@
# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other
# Spack Project Developers. See the top-level COPYRIGHT file for details.
#
# SPDX-License-Identifier: (Apache-2.0 OR MIT)
"""Manages the details on the images used in the build and the run stage."""
import json
import os.path
#: Global variable used to cache in memory the content of images.json
_data = None
def data():
"""Returns a dictionary with the static data on the images.
The dictionary is read from a JSON file lazily the first time
this function is called.
"""
global _data
if not _data:
json_dir = os.path.abspath(os.path.dirname(__file__))
json_file = os.path.join(json_dir, 'images.json')
with open(json_file) as f:
_data = json.load(f)
return _data
def build_info(image, spack_version):
"""Returns the name of the build image and its tag.
Args:
image (str): image to be used at run-time. Should be of the form
<image_name>:<image_tag> e.g. "ubuntu:18.04"
spack_version (str): version of Spack that we want to use to build
Returns:
A tuple with (image_name, image_tag) for the build image
"""
# Don't handle error here, as a wrong image should have been
# caught by the JSON schema
image_data = data()[image]
build_image = image_data['build']
# Try to check if we have a tag for this Spack version
try:
build_tag = image_data['build_tags'][spack_version]
except KeyError:
msg = ('the image "{0}" has no tag for Spack version "{1}" '
'[valid versions are {2}]')
msg = msg.format(build_image, spack_version,
', '.join(image_data['build_tags'].keys()))
raise ValueError(msg)
return build_image, build_tag
def package_info(image):
"""Returns the commands used to update system repositories, install
system packages and clean afterwards.
Args:
image (str): image to be used at run-time. Should be of the form
<image_name>:<image_tag> e.g. "ubuntu:18.04"
Returns:
A tuple of (update, install, clean) commands.
"""
image_data = data()[image]
update = image_data['update']
install = image_data['install']
clean = image_data['clean']
return update, install, clean

154
lib/spack/spack/container/writers/__init__.py Normal file
View File

@@ -0,0 +1,154 @@
# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other
# Spack Project Developers. See the top-level COPYRIGHT file for details.
#
# SPDX-License-Identifier: (Apache-2.0 OR MIT)
"""Writers for different kind of recipes and related
convenience functions.
"""
import collections
import copy
import spack.environment
import spack.schema.env
import spack.tengine as tengine
import spack.util.spack_yaml as syaml
from spack.container.images import build_info, package_info
#: Caches all the writers that are currently supported
_writer_factory = {}
def writer(name):
"""Decorator to register a factory for a recipe writer.
Each factory should take a configuration dictionary and return a
properly configured writer that, when called, prints the
corresponding recipe.
"""
def _decorator(factory):
_writer_factory[name] = factory
return factory
return _decorator
def create(configuration):
"""Returns a writer that conforms to the configuration passed as input.
Args:
configuration: how to generate the current recipe
"""
name = spack.environment.config_dict(configuration)['container']['format']
return _writer_factory[name](configuration)
def recipe(configuration):
"""Returns a recipe that conforms to the configuration passed as input.
Args:
configuration: how to generate the current recipe
"""
return create(configuration)()
class PathContext(tengine.Context):
"""Generic context used to instantiate templates of recipes that
install software in a common location and make it available
directly via PATH.
"""
def __init__(self, config):
self.config = spack.environment.config_dict(config)
self.container_config = self.config['container']
@tengine.context_property
def run(self):
"""Information related to the run image."""
image = self.container_config['base']['image']
Run = collections.namedtuple('Run', ['image'])
return Run(image=image)
@tengine.context_property
def build(self):
"""Information related to the build image."""
# Map the final image to the correct build image
run_image = self.container_config['base']['image']
spack_version = self.container_config['base']['spack']
image, tag = build_info(run_image, spack_version)
Build = collections.namedtuple('Build', ['image', 'tag'])
return Build(image=image, tag=tag)
@tengine.context_property
def strip(self):
"""Whether or not to strip binaries in the image"""
return self.container_config.get('strip', True)
@tengine.context_property
def paths(self):
"""Important paths in the image"""
Paths = collections.namedtuple('Paths', [
'environment', 'store', 'view'
])
return Paths(
environment='/opt/spack-environment',
store='/opt/software',
view='/opt/view'
)
@tengine.context_property
def manifest(self):
"""The spack.yaml file that should be used in the image"""
import jsonschema
# Copy in the part of spack.yaml prescribed in the configuration file
manifest = copy.deepcopy(self.config)
manifest.pop('container')
# Ensure that a few paths are where they need to be
manifest.setdefault('config', syaml.syaml_dict())
manifest['config']['install_tree'] = self.paths.store
manifest['view'] = self.paths.view
manifest = {'spack': manifest}
# Validate the manifest file
jsonschema.validate(manifest, schema=spack.schema.env.schema)
return syaml.dump(manifest, default_flow_style=False).strip()
@tengine.context_property
def os_packages(self):
"""Additional system packages that are needed at run-time."""
package_list = self.container_config.get('os_packages', None)
if not package_list:
return package_list
image = self.container_config['base']['image']
update, install, clean = package_info(image)
Packages = collections.namedtuple(
'Packages', ['update', 'install', 'list', 'clean']
)
return Packages(update=update, install=install,
list=package_list, clean=clean)
@tengine.context_property
def extra_instructions(self):
Extras = collections.namedtuple('Extra', ['build', 'final'])
extras = self.container_config.get('extra_instructions', {})
build, final = extras.get('build', None), extras.get('final', None)
return Extras(build=build, final=final)
@tengine.context_property
def labels(self):
return self.container_config.get('labels', {})
def __call__(self):
"""Returns the recipe as a string"""
env = tengine.make_environment()
t = env.get_template(self.template_name)
return t.render(**self.to_dict())
# Import after function definition all the modules in this package,
# so that registration of writers will happen automatically
import spack.container.writers.singularity # noqa
import spack.container.writers.docker # noqa

30
lib/spack/spack/container/writers/docker.py Normal file
View File

@@ -0,0 +1,30 @@
# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other
# Spack Project Developers. See the top-level COPYRIGHT file for details.
#
# SPDX-License-Identifier: (Apache-2.0 OR MIT)
import spack.tengine as tengine
from . import writer, PathContext
@writer('docker')
class DockerContext(PathContext):
"""Context used to instantiate a Dockerfile"""
#: Name of the template used for Dockerfiles
template_name = 'container/Dockerfile'
@tengine.context_property
def manifest(self):
manifest_str = super(DockerContext, self).manifest
# Docker doesn't support HEREDOC so we need to resort to
# a horrible echo trick to have the manifest in the Dockerfile
echoed_lines = []
for idx, line in enumerate(manifest_str.split('\n')):
if idx == 0:
echoed_lines.append('&& (echo "' + line + '" \\')
continue
echoed_lines.append('&& echo "' + line + '" \\')
echoed_lines[-1] = echoed_lines[-1].replace(' \\', ')')
return '\n'.join(echoed_lines)

33
lib/spack/spack/container/writers/singularity.py Normal file
View File

@@ -0,0 +1,33 @@
# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other
# Spack Project Developers. See the top-level COPYRIGHT file for details.
#
# SPDX-License-Identifier: (Apache-2.0 OR MIT)
import spack.tengine as tengine
from . import writer, PathContext
@writer('singularity')
class SingularityContext(PathContext):
"""Context used to instantiate a Singularity definition file"""
#: Name of the template used for Singularity definition files
template_name = 'container/singularity.def'
@property
def singularity_config(self):
return self.container_config.get('singularity', {})
@tengine.context_property
def runscript(self):
return self.singularity_config.get('runscript', '')
@tengine.context_property
def startscript(self):
return self.singularity_config.get('startscript', '')
@tengine.context_property
def test(self):
return self.singularity_config.get('test', '')
@tengine.context_property
def help(self):
return self.singularity_config.get('help', '')

82
lib/spack/spack/schema/container.py Normal file
View File

@@ -0,0 +1,82 @@
# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other
# Spack Project Developers. See the top-level COPYRIGHT file for details.
#
# SPDX-License-Identifier: (Apache-2.0 OR MIT)
"""Schema for the 'container' subsection of Spack environments."""
#: Schema for the container attribute included in Spack environments
container_schema = {
'type': 'object',
'additionalProperties': False,
'properties': {
# The recipe formats that are currently supported by the command
'format': {
'type': 'string',
'enum': ['docker', 'singularity']
},
# Describes the base image to start from and the version
# of Spack to be used
'base': {
'type': 'object',
'additionalProperties': False,
'properties': {
'image': {
'type': 'string',
'enum': ['ubuntu:18.04',
'ubuntu:16.04',
'centos:7',
'centos:6']
},
'spack': {
'type': 'string',
'enum': ['develop', '0.14', '0.14.0']
}
},
'required': ['image', 'spack']
},
# Whether or not to strip installed binaries
'strip': {
'type': 'boolean',
'default': True
},
# Additional system packages that are needed at runtime
'os_packages': {
'type': 'array',
'items': {
'type': 'string'
}
},
# Add labels to the image
'labels': {
'type': 'object',
},
# Add a custom extra section at the bottom of a stage
'extra_instructions': {
'type': 'object',
'additionalProperties': False,
'properties': {
'build': {'type': 'string'},
'final': {'type': 'string'}
}
},
# Reserved for properties that are specific to each format
'singularity': {
'type': 'object',
'additionalProperties': False,
'default': {},
'properties': {
'runscript': {'type': 'string'},
'startscript': {'type': 'string'},
'test': {'type': 'string'},
'help': {'type': 'string'}
}
},
'docker': {
'type': 'object',
'additionalProperties': False,
'default': {},
}
}
}
properties = {'container': container_schema}

2
lib/spack/spack/schema/merged.py
View File

@@ -13,6 +13,7 @@
import spack.schema.cdash
import spack.schema.compilers
import spack.schema.config
import spack.schema.container
import spack.schema.gitlab_ci
import spack.schema.mirrors
import spack.schema.modules
@@ -26,6 +27,7 @@
spack.schema.cdash.properties,
spack.schema.compilers.properties,
spack.schema.config.properties,
spack.schema.container.properties,
spack.schema.gitlab_ci.properties,
spack.schema.mirrors.properties,
spack.schema.modules.properties,

4
lib/spack/spack/test/cmd/gc.py
View File

@@ -30,7 +30,9 @@ def test_packages_are_removed(config, mutable_database, capsys):
@pytest.mark.db
def test_gc_with_environment(config, mutable_database, capsys):
def test_gc_with_environment(
config, mutable_database, mutable_mock_env_path, capsys
):
s = spack.spec.Spec('simple-inheritance')
s.concretize()
s.package.do_install(fake=True, explicit=True)

2
lib/spack/spack/test/cmd/test.py
View File

@@ -1,4 +1,4 @@
# Copyright 2013-2019 Lawrence Livermore National Security, LLC and other
# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other
# Spack Project Developers. See the top-level COPYRIGHT file for details.
#
# SPDX-License-Identifier: (Apache-2.0 OR MIT)

16
lib/spack/spack/test/container/cli.py Normal file
View File

@@ -0,0 +1,16 @@
# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other
# Spack Project Developers. See the top-level COPYRIGHT file for details.
#
# SPDX-License-Identifier: (Apache-2.0 OR MIT)
import llnl.util.filesystem as fs
import spack.main
containerize = spack.main.SpackCommand('containerize')
def test_command(configuration_dir, capsys):
with capsys.disabled():
with fs.working_dir(configuration_dir):
output = containerize()
assert 'FROM spack/ubuntu-bionic' in output

43
lib/spack/spack/test/container/conftest.py Normal file
View File

@@ -0,0 +1,43 @@
# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other
# Spack Project Developers. See the top-level COPYRIGHT file for details.
#
# SPDX-License-Identifier: (Apache-2.0 OR MIT)
import pytest
import spack.util.spack_yaml as syaml
@pytest.fixture()
def minimal_configuration():
return {
'spack': {
'specs': [
'gromacs',
'mpich',
'fftw precision=float'
],
'container': {
'format': 'docker',
'base': {
'image': 'ubuntu:18.04',
'spack': 'develop'
}
}
}
}
@pytest.fixture()
def config_dumper(tmpdir):
"""Function that dumps an environment config in a temporary folder."""
def dumper(configuration):
content = syaml.dump(configuration, default_flow_style=False)
config_file = tmpdir / 'spack.yaml'
config_file.write(content)
return str(tmpdir)
return dumper
@pytest.fixture()
def configuration_dir(minimal_configuration, config_dumper):
return config_dumper(minimal_configuration)

74
lib/spack/spack/test/container/docker.py Normal file
View File

@@ -0,0 +1,74 @@
# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other
# Spack Project Developers. See the top-level COPYRIGHT file for details.
#
# SPDX-License-Identifier: (Apache-2.0 OR MIT)
import spack.container.writers as writers
def test_manifest(minimal_configuration):
writer = writers.create(minimal_configuration)
manifest_str = writer.manifest
for line in manifest_str.split('\n'):
assert 'echo' in line
def test_build_and_run_images(minimal_configuration):
writer = writers.create(minimal_configuration)
# Test the output of run property
run = writer.run
assert run.image == 'ubuntu:18.04'
# Test the output of the build property
build = writer.build
assert build.image == 'spack/ubuntu-bionic'
assert build.tag == 'latest'
def test_packages(minimal_configuration):
# In this minimal configuration we don't have packages
writer = writers.create(minimal_configuration)
assert writer.os_packages is None
# If we add them a list should be returned
pkgs = ['libgomp1']
minimal_configuration['spack']['container']['os_packages'] = pkgs
writer = writers.create(minimal_configuration)
p = writer.os_packages
assert p.update
assert p.install
assert p.clean
assert p.list == pkgs
def test_ensure_render_works(minimal_configuration):
# Here we just want to ensure that nothing is raised
writer = writers.create(minimal_configuration)
writer()
def test_strip_is_set_from_config(minimal_configuration):
writer = writers.create(minimal_configuration)
assert writer.strip is True
minimal_configuration['spack']['container']['strip'] = False
writer = writers.create(minimal_configuration)
assert writer.strip is False
def test_extra_instructions_is_set_from_config(minimal_configuration):
writer = writers.create(minimal_configuration)
assert writer.extra_instructions == (None, None)
test_line = 'RUN echo Hello world!'
e = minimal_configuration['spack']['container']
e['extra_instructions'] = {}
e['extra_instructions']['build'] = test_line
writer = writers.create(minimal_configuration)
assert writer.extra_instructions == (test_line, None)
e['extra_instructions']['final'] = test_line
del e['extra_instructions']['build']
writer = writers.create(minimal_configuration)
assert writer.extra_instructions == (None, test_line)

58
lib/spack/spack/test/container/images.py Normal file
View File

@@ -0,0 +1,58 @@
# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other
# Spack Project Developers. See the top-level COPYRIGHT file for details.
#
# SPDX-License-Identifier: (Apache-2.0 OR MIT)
import os.path
import pytest
import spack.container
@pytest.mark.parametrize('image,spack_version,expected', [
('ubuntu:18.04', 'develop', ('spack/ubuntu-bionic', 'latest')),
('ubuntu:18.04', '0.14.0', ('spack/ubuntu-bionic', '0.14.0')),
])
def test_build_info(image, spack_version, expected):
output = spack.container.images.build_info(image, spack_version)
assert output == expected
@pytest.mark.parametrize('image,spack_version', [
('ubuntu:18.04', 'doesnotexist')
])
def test_build_info_error(image, spack_version):
with pytest.raises(ValueError, match=r"has no tag for"):
spack.container.images.build_info(image, spack_version)
@pytest.mark.parametrize('image', [
'ubuntu:18.04'
])
def test_package_info(image):
update, install, clean = spack.container.images.package_info(image)
assert update
assert install
assert clean
@pytest.mark.parametrize('extra_config,expected_msg', [
({'modules': {'enable': ['tcl']}}, 'the subsection "modules" in'),
({'concretization': 'separately'}, 'the "concretization" attribute'),
({'config': {'install_tree': '/some/dir'}},
'the "config:install_tree" attribute has been set'),
({'view': '/some/dir'}, 'the "view" attribute has been set')
])
def test_validate(
extra_config, expected_msg, minimal_configuration, config_dumper
):
minimal_configuration['spack'].update(extra_config)
spack_yaml_dir = config_dumper(minimal_configuration)
spack_yaml = os.path.join(spack_yaml_dir, 'spack.yaml')
with pytest.warns(UserWarning) as w:
spack.container.validate(spack_yaml)
# Tests are designed to raise only one warning
assert len(w) == 1
assert expected_msg in str(w.pop().message)

16
lib/spack/spack/test/container/schema.py Normal file
View File

@@ -0,0 +1,16 @@
# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other
# Spack Project Developers. See the top-level COPYRIGHT file for details.
#
# SPDX-License-Identifier: (Apache-2.0 OR MIT)
import spack.container
import spack.schema.container
def test_images_in_schema():
properties = spack.schema.container.container_schema['properties']
allowed_images = set(
properties['base']['properties']['image']['enum']
)
images_in_json = set(x for x in spack.container.images.data())
assert images_in_json == allowed_images

42
lib/spack/spack/test/container/singularity.py Normal file
View File

@@ -0,0 +1,42 @@
# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other
# Spack Project Developers. See the top-level COPYRIGHT file for details.
#
# SPDX-License-Identifier: (Apache-2.0 OR MIT)
import pytest
import spack.container.writers as writers
@pytest.fixture
def singularity_configuration(minimal_configuration):
minimal_configuration['spack']['container']['format'] = 'singularity'
return minimal_configuration
def test_ensure_render_works(singularity_configuration):
container_config = singularity_configuration['spack']['container']
assert container_config['format'] == 'singularity'
# Here we just want to ensure that nothing is raised
writer = writers.create(singularity_configuration)
writer()
@pytest.mark.parametrize('properties,expected', [
({'runscript': '/opt/view/bin/h5ls'},
{'runscript': '/opt/view/bin/h5ls',
'startscript': '',
'test': '',
'help': ''})
])
def test_singularity_specific_properties(
properties, expected, singularity_configuration
):
# Set the property in the configuration
container_config = singularity_configuration['spack']['container']
for name, value in properties.items():
container_config.setdefault('singularity', {})[name] = value
# Assert the properties return the expected values
writer = writers.create(singularity_configuration)
for name, value in expected.items():
assert getattr(writer, name) == value

6
share/spack/spack-completion.bash
View File

@@ -313,7 +313,7 @@ _spack() {
then
SPACK_COMPREPLY="-h --help -H --all-help --color -C --config-scope -d --debug --timestamp --pdb -e --env -D --env-dir -E --no-env --use-env-repo -k --insecure -l --enable-locks -L --disable-locks -m --mock -p --profile --sorted-profile --lines -v --verbose --stacktrace -V --version --print-shell-vars"
else
SPACK_COMPREPLY="activate add arch blame bootstrap build build-env buildcache cd checksum ci clean clone commands compiler compilers concretize config configure create deactivate debug dependencies dependents deprecate dev-build diy docs edit env extensions fetch find flake8 gc gpg graph help info install license list load location log-parse maintainers mirror module patch pkg providers pydoc python reindex remove rm repo resource restage setup spec stage test uninstall unload upload-s3 url verify versions view"
SPACK_COMPREPLY="activate add arch blame bootstrap build build-env buildcache cd checksum ci clean clone commands compiler compilers concretize config configure containerize create deactivate debug dependencies dependents deprecate dev-build diy docs edit env extensions fetch find flake8 gc gpg graph help info install license list load location log-parse maintainers mirror module patch pkg providers pydoc python reindex remove rm repo resource restage setup spec stage test uninstall unload upload-s3 url verify versions view"
fi
}
@@ -628,6 +628,10 @@ _spack_configure() {
fi
}
_spack_containerize() {
SPACK_COMPREPLY="-h --help"
}
_spack_create() {
if $list_options
then

51
share/spack/templates/container/Dockerfile Normal file
View File

@@ -0,0 +1,51 @@
# Build stage with Spack pre-installed and ready to be used
FROM {{ build.image }}:{{ build.tag }} as builder
# What we want to install and how we want to install it
# is specified in a manifest file (spack.yaml)
RUN mkdir {{ paths.environment }} \
{{ manifest }} > {{ paths.environment }}/spack.yaml
# Install the software, remove unecessary deps
RUN cd {{ paths.environment }} && spack install && spack gc -y
{% if strip %}
# Strip all the binaries
RUN find -L {{ paths.view }}/* -type f -exec readlink -f '{}' \; | \
xargs file -i | \
grep 'charset=binary' | \
grep 'x-executable\|x-archive\|x-sharedlib' | \
awk -F: '{print $1}' | xargs strip -s
{% endif %}
# Modifications to the environment that are necessary to run
RUN cd {{ paths.environment }} && \
spack env activate --sh -d . >> /etc/profile.d/z10_spack_environment.sh
{% if extra_instructions.build %}
{{ extra_instructions.build }}
{% endif %}
# Bare OS image to run the installed executables
FROM {{ run.image }}
COPY --from=builder {{ paths.environment }} {{ paths.environment }}
COPY --from=builder {{ paths.store }} {{ paths.store }}
COPY --from=builder {{ paths.view }} {{ paths.view }}
COPY --from=builder /etc/profile.d/z10_spack_environment.sh /etc/profile.d/z10_spack_environment.sh
{% if os_packages %}
RUN {{ os_packages.update }} \
&& {{ os_packages.install }}{% for pkg in os_packages.list %} {{ pkg }}{% endfor %} \
&& {{ os_packages.clean }}
{% endif %}
{% if extra_instructions.final %}
{{ extra_instructions.final }}
{% endif %}
{% for label, value in labels.items() %}
LABEL "{{ label }}"="{{ value }}"
{% endfor %}
ENTRYPOINT ["/bin/bash", "--rcfile", "/etc/profile", "-l"]

90
share/spack/templates/container/singularity.def Normal file
View File

@@ -0,0 +1,90 @@
Bootstrap: docker
From: {{ build.image }}:{{ build.tag }}
Stage: build
%post
# Create the manifest file for the installation in /opt/spack-environment
mkdir {{ paths.environment }} && cd {{ paths.environment }}
cat << EOF > spack.yaml
{{ manifest }}
EOF
# Install all the required software
. /opt/spack/share/spack/setup-env.sh
spack install
spack gc -y
spack env activate --sh -d . >> {{ paths.environment }}/environment_modifications.sh
{% if strip %}
# Strip the binaries to reduce the size of the image
find -L {{ paths.view }}/* -type f -exec readlink -f '{}' \; | \
xargs file -i | \
grep 'charset=binary' | \
grep 'x-executable\|x-archive\|x-sharedlib' | \
awk -F: '{print $1}' | xargs strip -s
{% endif %}
{% if extra_instructions.build %}
{{ extra_instructions.build }}
{% endif %}
{% if apps %}
{% for application, help_text in apps.items() %}
%apprun {{ application }}
exec /opt/view/bin/{{ application }} "$@"
%apphelp {{ application }}
{{help_text }}
{% endfor %}
{% endif %}
Bootstrap: docker
From: {{ run.image }}
Stage: final
%files from build
{{ paths.environment }} /opt
{{ paths.store }} /opt
{{ paths.view }} /opt
{{ paths.environment }}/environment_modifications.sh {{ paths.environment }}/environment_modifications.sh
%post
{% if os_packages.list %}
# Update, install and cleanup of system packages
{{ os_packages.update }}
{{ os_packages.install }} {{ os_packages.list | join | replace('\n', ' ') }}
{{ os_packages.clean }}
{% endif %}
# Modify the environment without relying on sourcing shell specific files at startup
cat {{ paths.environment }}/environment_modifications.sh >> $SINGULARITY_ENVIRONMENT
{% if extra_instructions.final %}
{{ extra_instructions.final }}
{% endif %}
{% if runscript %}
%runscript
{{ runscript }}
{% endif %}
{% if startscript %}
%startscript
{{ startscript }}
{% endif %}
{% if test %}
%test
{{ test }}
{% endif %}
{% if help %}
%help
{{ help }}
{% endif %}
{% if labels %}
%labels
{% for label, value in labels.items() %}
{{ label }} {{ value }}
{% endfor %}
{% endif %}