spack containerize generates containers from envs (#14202)
This PR adds a new command to Spack:
```console
$ spack containerize -h
usage: spack containerize [-h] [--config CONFIG]
creates recipes to build images for different container runtimes
optional arguments:
-h, --help show this help message and exit
--config CONFIG configuration for the container recipe that will be generated
```
which takes an environment with an additional `container` section:
```yaml
spack:
specs:
- gromacs build_type=Release
- mpich
- fftw precision=float
packages:
all:
target: [broadwell]
container:
# Select the format of the recipe e.g. docker,
# singularity or anything else that is currently supported
format: docker
# Select from a valid list of images
base:
image: "ubuntu:18.04"
spack: prerelease
# Additional system packages that are needed at runtime
os_packages:
- libgomp1
```
and turns it into a `Dockerfile` or a Singularity definition file, for instance:
```Dockerfile
# Build stage with Spack pre-installed and ready to be used
FROM spack/ubuntu-bionic:prerelease as builder
# What we want to install and how we want to install it
# is specified in a manifest file (spack.yaml)
RUN mkdir /opt/spack-environment \
&& (echo "spack:" \
&& echo " specs:" \
&& echo " - gromacs build_type=Release" \
&& echo " - mpich" \
&& echo " - fftw precision=float" \
&& echo " packages:" \
&& echo " all:" \
&& echo " target:" \
&& echo " - broadwell" \
&& echo " config:" \
&& echo " install_tree: /opt/software" \
&& echo " concretization: together" \
&& echo " view: /opt/view") > /opt/spack-environment/spack.yaml
# Install the software, remove unecessary deps and strip executables
RUN cd /opt/spack-environment && spack install && spack autoremove -y
RUN find -L /opt/view/* -type f -exec readlink -f '{}' \; | \
xargs file -i | \
grep 'charset=binary' | \
grep 'x-executable\|x-archive\|x-sharedlib' | \
awk -F: '{print $1}' | xargs strip -s
# Modifications to the environment that are necessary to run
RUN cd /opt/spack-environment && \
spack env activate --sh -d . >> /etc/profile.d/z10_spack_environment.sh
# Bare OS image to run the installed executables
FROM ubuntu:18.04
COPY --from=builder /opt/spack-environment /opt/spack-environment
COPY --from=builder /opt/software /opt/software
COPY --from=builder /opt/view /opt/view
COPY --from=builder /etc/profile.d/z10_spack_environment.sh /etc/profile.d/z10_spack_environment.sh
RUN apt-get -yqq update && apt-get -yqq upgrade \
&& apt-get -yqq install libgomp1 \
&& rm -rf /var/lib/apt/lists/*
ENTRYPOINT ["/bin/bash", "--rcfile", "/etc/profile", "-l"]
```
This commit is contained in:
Massimiliano Culpo
GitHub
parent
ed501eaab2
commit
9635ff3d20
307
lib/spack/docs/containers.rst
Normal file
307
lib/spack/docs/containers.rst
Normal file
@ -0,0 +1,307 @@
|
||||
.. Copyright 2013-2020 Lawrence Livermore National Security, LLC and other
|
||||
Spack Project Developers. See the top-level COPYRIGHT file for details.
|
||||
|
||||
SPDX-License-Identifier: (Apache-2.0 OR MIT)
|
||||
|
||||
.. _containers:
|
||||
|
||||
================
|
||||
Container Images
|
||||
================
|
||||
|
||||
Spack can be an ideal tool to setup images for containers since all the
|
||||
features discussed in :ref:`environments` can greatly help to manage
|
||||
the installation of software during the image build process. Nonetheless,
|
||||
building a production image from scratch still requires a lot of
|
||||
boilerplate to:
|
||||
|
||||
- Get Spack working within the image, possibly running as root
|
||||
- Minimize the physical size of the software installed
|
||||
- Properly update the system software in the base image
|
||||
|
||||
To facilitate users with these tedious tasks, Spack provides a command
|
||||
to automatically generate recipes for container images based on
|
||||
Environments:
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
$ ls
|
||||
spack.yaml
|
||||
|
||||
$ spack containerize
|
||||
# Build stage with Spack pre-installed and ready to be used
|
||||
FROM spack/centos7:latest as builder
|
||||
|
||||
# What we want to install and how we want to install it
|
||||
# is specified in a manifest file (spack.yaml)
|
||||
RUN mkdir /opt/spack-environment \
|
||||
&& (echo "spack:" \
|
||||
&& echo " specs:" \
|
||||
&& echo " - gromacs+mpi" \
|
||||
&& echo " - mpich" \
|
||||
&& echo " concretization: together" \
|
||||
&& echo " config:" \
|
||||
&& echo " install_tree: /opt/software" \
|
||||
&& echo " view: /opt/view") > /opt/spack-environment/spack.yaml
|
||||
|
||||
# Install the software, remove unecessary deps
|
||||
RUN cd /opt/spack-environment && spack install && spack gc -y
|
||||
|
||||
# Strip all the binaries
|
||||
RUN find -L /opt/view/* -type f -exec readlink -f '{}' \; | \
|
||||
xargs file -i | \
|
||||
grep 'charset=binary' | \
|
||||
grep 'x-executable\|x-archive\|x-sharedlib' | \
|
||||
awk -F: '{print $1}' | xargs strip -s
|
||||
|
||||
# Modifications to the environment that are necessary to run
|
||||
RUN cd /opt/spack-environment && \
|
||||
spack env activate --sh -d . >> /etc/profile.d/z10_spack_environment.sh
|
||||
|
||||
|
||||
# Bare OS image to run the installed executables
|
||||
FROM centos:7
|
||||
|
||||
COPY --from=builder /opt/spack-environment /opt/spack-environment
|
||||
COPY --from=builder /opt/software /opt/software
|
||||
COPY --from=builder /opt/view /opt/view
|
||||
COPY --from=builder /etc/profile.d/z10_spack_environment.sh /etc/profile.d/z10_spack_environment.sh
|
||||
|
||||
RUN yum update -y && yum install -y epel-release && yum update -y \
|
||||
&& yum install -y libgomp \
|
||||
&& rm -rf /var/cache/yum && yum clean all
|
||||
|
||||
RUN echo 'export PS1="\[$(tput bold)\]\[$(tput setaf 1)\][gromacs]\[$(tput setaf 2)\]\u\[$(tput sgr0)\]:\w $ \[$(tput sgr0)\]"' >> ~/.bashrc
|
||||
|
||||
|
||||
LABEL "app"="gromacs"
|
||||
LABEL "mpi"="mpich"
|
||||
|
||||
ENTRYPOINT ["/bin/bash", "--rcfile", "/etc/profile", "-l"]
|
||||
|
||||
|
||||
The bits that make this automation possible are discussed in details
|
||||
below. All the images generated in this way will be based on
|
||||
multi-stage builds with:
|
||||
|
||||
- A fat ``build`` stage containing common build tools and Spack itself
|
||||
- A minimal ``final`` stage containing only the software requested by the user
|
||||
|
||||
-----------------
|
||||
Spack Base Images
|
||||
-----------------
|
||||
|
||||
Docker images with Spack preinstalled and ready to be used are
|
||||
built on `Docker Hub <https://hub.docker.com/u/spack>`_
|
||||
at every push to ``develop`` or to a release branch. The OS that
|
||||
are currently supported are summarized in the table below:
|
||||
|
||||
.. _containers-supported-os:
|
||||
|
||||
.. list-table:: Supported operating systems
|
||||
:header-rows: 1
|
||||
|
||||
* - Operating System
|
||||
- Base Image
|
||||
- Spack Image
|
||||
* - Ubuntu 16.04
|
||||
- ``ubuntu:16.04``
|
||||
- ``spack/ubuntu-xenial``
|
||||
* - Ubuntu 18.04
|
||||
- ``ubuntu:16.04``
|
||||
- ``spack/ubuntu-bionic``
|
||||
* - CentOS 6
|
||||
- ``centos:6``
|
||||
- ``spack/centos6``
|
||||
* - CentOS 7
|
||||
- ``centos:7``
|
||||
- ``spack/centos7``
|
||||
|
||||
All the images are tagged with the corresponding release of Spack:
|
||||
|
||||
.. image:: dockerhub_spack.png
|
||||
|
||||
with the exception of the ``latest`` tag that points to the HEAD
|
||||
of the ``develop`` branch. These images are available for anyone
|
||||
to use and take care of all the repetitive tasks that are necessary
|
||||
to setup Spack within a container. All the container recipes generated
|
||||
automatically by Spack use them as base images for their ``build`` stage.
|
||||
|
||||
|
||||
-------------------------
|
||||
Environment Configuration
|
||||
-------------------------
|
||||
|
||||
Any Spack Environment can be used for the automatic generation of container
|
||||
recipes. Sensible defaults are provided for things like the base image or the
|
||||
version of Spack used in the image. If a finer tuning is needed it can be
|
||||
obtained by adding the relevant metadata under the ``container`` attribute
|
||||
of environments:
|
||||
|
||||
.. code-block:: yaml
|
||||
|
||||
spack:
|
||||
specs:
|
||||
- gromacs+mpi
|
||||
- mpich
|
||||
|
||||
container:
|
||||
# Select the format of the recipe e.g. docker,
|
||||
# singularity or anything else that is currently supported
|
||||
format: docker
|
||||
|
||||
# Select from a valid list of images
|
||||
base:
|
||||
image: "centos:7"
|
||||
spack: develop
|
||||
|
||||
# Whether or not to strip binaries
|
||||
strip: true
|
||||
|
||||
# Additional system packages that are needed at runtime
|
||||
os_packages:
|
||||
- libgomp
|
||||
|
||||
# Extra instructions
|
||||
extra_instructions:
|
||||
final: |
|
||||
RUN echo 'export PS1="\[$(tput bold)\]\[$(tput setaf 1)\][gromacs]\[$(tput setaf 2)\]\u\[$(tput sgr0)\]:\w $ \[$(tput sgr0)\]"' >> ~/.bashrc
|
||||
|
||||
# Labels for the image
|
||||
labels:
|
||||
app: "gromacs"
|
||||
mpi: "mpich"
|
||||
|
||||
The tables below describe the configuration options that are currently supported:
|
||||
|
||||
.. list-table:: General configuration options for the ``container`` section of ``spack.yaml``
|
||||
:header-rows: 1
|
||||
|
||||
* - Option Name
|
||||
- Description
|
||||
- Allowed Values
|
||||
- Required
|
||||
* - ``format``
|
||||
- The format of the recipe
|
||||
- ``docker`` or ``singularity``
|
||||
- Yes
|
||||
* - ``base:image``
|
||||
- Base image for ``final`` stage
|
||||
- See :ref:`containers-supported-os`
|
||||
- Yes
|
||||
* - ``base:spack``
|
||||
- Version of Spack
|
||||
- Valid tags for ``base:image``
|
||||
- Yes
|
||||
* - ``strip``
|
||||
- Whether to strip binaries
|
||||
- ``true`` (default) or ``false``
|
||||
- No
|
||||
* - ``os_packages``
|
||||
- System packages to be installed
|
||||
- Valid packages for the ``final`` OS
|
||||
- No
|
||||
* - ``extra_instructions:build``
|
||||
- Extra instructions (e.g. `RUN`, `COPY`, etc.) at the end of the ``build`` stage
|
||||
- Anything understood by the current ``format``
|
||||
- No
|
||||
* - ``extra_instructions:final``
|
||||
- Extra instructions (e.g. `RUN`, `COPY`, etc.) at the end of the ``final`` stage
|
||||
- Anything understood by the current ``format``
|
||||
- No
|
||||
* - ``labels``
|
||||
- Labels to tag the image
|
||||
- Pairs of key-value strings
|
||||
- No
|
||||
|
||||
.. list-table:: Configuration options specific to Singularity
|
||||
:header-rows: 1
|
||||
|
||||
* - Option Name
|
||||
- Description
|
||||
- Allowed Values
|
||||
- Required
|
||||
* - ``singularity:runscript``
|
||||
- Content of ``%runscript``
|
||||
- Any valid script
|
||||
- No
|
||||
* - ``singularity:startscript``
|
||||
- Content of ``%startscript``
|
||||
- Any valid script
|
||||
- No
|
||||
* - ``singularity:test``
|
||||
- Content of ``%test``
|
||||
- Any valid script
|
||||
- No
|
||||
* - ``singularity:help``
|
||||
- Description of the image
|
||||
- Description string
|
||||
- No
|
||||
|
||||
Once the Environment is properly configured a recipe for a container
|
||||
image can be printed to standard output by issuing the following
|
||||
command from the directory where the ``spack.yaml`` resides:
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
$ spack containerize
|
||||
|
||||
The example ``spack.yaml`` above would produce for instance the
|
||||
following ``Dockerfile``:
|
||||
|
||||
.. code-block:: docker
|
||||
|
||||
# Build stage with Spack pre-installed and ready to be used
|
||||
FROM spack/centos7:latest as builder
|
||||
|
||||
# What we want to install and how we want to install it
|
||||
# is specified in a manifest file (spack.yaml)
|
||||
RUN mkdir /opt/spack-environment \
|
||||
&& (echo "spack:" \
|
||||
&& echo " specs:" \
|
||||
&& echo " - gromacs+mpi" \
|
||||
&& echo " - mpich" \
|
||||
&& echo " concretization: together" \
|
||||
&& echo " config:" \
|
||||
&& echo " install_tree: /opt/software" \
|
||||
&& echo " view: /opt/view") > /opt/spack-environment/spack.yaml
|
||||
|
||||
# Install the software, remove unecessary deps
|
||||
RUN cd /opt/spack-environment && spack install && spack gc -y
|
||||
|
||||
# Strip all the binaries
|
||||
RUN find -L /opt/view/* -type f -exec readlink -f '{}' \; | \
|
||||
xargs file -i | \
|
||||
grep 'charset=binary' | \
|
||||
grep 'x-executable\|x-archive\|x-sharedlib' | \
|
||||
awk -F: '{print $1}' | xargs strip -s
|
||||
|
||||
# Modifications to the environment that are necessary to run
|
||||
RUN cd /opt/spack-environment && \
|
||||
spack env activate --sh -d . >> /etc/profile.d/z10_spack_environment.sh
|
||||
|
||||
|
||||
# Bare OS image to run the installed executables
|
||||
FROM centos:7
|
||||
|
||||
COPY --from=builder /opt/spack-environment /opt/spack-environment
|
||||
COPY --from=builder /opt/software /opt/software
|
||||
COPY --from=builder /opt/view /opt/view
|
||||
COPY --from=builder /etc/profile.d/z10_spack_environment.sh /etc/profile.d/z10_spack_environment.sh
|
||||
|
||||
RUN yum update -y && yum install -y epel-release && yum update -y \
|
||||
&& yum install -y libgomp \
|
||||
&& rm -rf /var/cache/yum && yum clean all
|
||||
|
||||
RUN echo 'export PS1="\[$(tput bold)\]\[$(tput setaf 1)\][gromacs]\[$(tput setaf 2)\]\u\[$(tput sgr0)\]:\w $ \[$(tput sgr0)\]"' >> ~/.bashrc
|
||||
|
||||
|
||||
LABEL "app"="gromacs"
|
||||
LABEL "mpi"="mpich"
|
||||
|
||||
ENTRYPOINT ["/bin/bash", "--rcfile", "/etc/profile", "-l"]
|
||||
|
||||
.. note::
|
||||
Spack can also produce Singularity definition files to build the image. The
|
||||
minimum version of Singularity required to build a SIF (Singularity Image Format)
|
||||
from them is ``3.5.3``.
|
||||
BIN
lib/spack/docs/dockerhub_spack.png
Normal file
BIN
lib/spack/docs/dockerhub_spack.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 88 KiB |
@ -49,6 +49,8 @@ Spack uses a "manifest and lock" model similar to `Bundler gemfiles
|
||||
managers. The user input file is named ``spack.yaml`` and the lock
|
||||
file is named ``spack.lock``
|
||||
|
||||
.. _environments-using:
|
||||
|
||||
------------------
|
||||
Using Environments
|
||||
------------------
|
||||
|
||||
@ -66,6 +66,7 @@ or refer to the full manual below.
|
||||
config_yaml
|
||||
build_settings
|
||||
environments
|
||||
containers
|
||||
mirrors
|
||||
module_file_support
|
||||
repositories
|
||||
|
||||
25
lib/spack/spack/cmd/containerize.py
Normal file
25
lib/spack/spack/cmd/containerize.py
Normal file
@ -0,0 +1,25 @@
|
||||
# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other
|
||||
# Spack Project Developers. See the top-level COPYRIGHT file for details.
|
||||
#
|
||||
# SPDX-License-Identifier: (Apache-2.0 OR MIT)
|
||||
import os
|
||||
import os.path
|
||||
import spack.container
|
||||
|
||||
description = ("creates recipes to build images for different"
|
||||
" container runtimes")
|
||||
section = "container"
|
||||
level = "long"
|
||||
|
||||
|
||||
def containerize(parser, args):
|
||||
config_dir = args.env_dir or os.getcwd()
|
||||
config_file = os.path.abspath(os.path.join(config_dir, 'spack.yaml'))
|
||||
if not os.path.exists(config_file):
|
||||
msg = 'file not found: {0}'
|
||||
raise ValueError(msg.format(config_file))
|
||||
|
||||
config = spack.container.validate(config_file)
|
||||
|
||||
recipe = spack.container.recipe(config)
|
||||
print(recipe)
|
||||
81
lib/spack/spack/container/__init__.py
Normal file
81
lib/spack/spack/container/__init__.py
Normal file
@ -0,0 +1,81 @@
|
||||
# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other
|
||||
# Spack Project Developers. See the top-level COPYRIGHT file for details.
|
||||
#
|
||||
# SPDX-License-Identifier: (Apache-2.0 OR MIT)
|
||||
"""Package that provides functions and classes to
|
||||
generate container recipes from a Spack environment
|
||||
"""
|
||||
import warnings
|
||||
|
||||
import spack.environment
|
||||
import spack.schema.env as env
|
||||
import spack.util.spack_yaml as syaml
|
||||
from .writers import recipe
|
||||
|
||||
__all__ = ['validate', 'recipe']
|
||||
|
||||
|
||||
def validate(configuration_file):
|
||||
"""Validate a Spack environment YAML file that is being used to generate a
|
||||
recipe for a container.
|
||||
|
||||
Since a few attributes of the configuration must have specific values for
|
||||
the container recipe, this function returns a sanitized copy of the
|
||||
configuration in the input file. If any modification is needed, a warning
|
||||
will be issued.
|
||||
|
||||
Args:
|
||||
configuration_file (str): path to the Spack environment YAML file
|
||||
|
||||
Returns:
|
||||
A sanitized copy of the configuration stored in the input file
|
||||
"""
|
||||
import jsonschema
|
||||
with open(configuration_file) as f:
|
||||
config = syaml.load(f)
|
||||
|
||||
# Ensure we have a "container" attribute with sensible defaults set
|
||||
env_dict = spack.environment.config_dict(config)
|
||||
env_dict.setdefault('container', {
|
||||
'format': 'docker',
|
||||
'base': {'image': 'ubuntu:18.04', 'spack': 'develop'}
|
||||
})
|
||||
env_dict['container'].setdefault('format', 'docker')
|
||||
env_dict['container'].setdefault(
|
||||
'base', {'image': 'ubuntu:18.04', 'spack': 'develop'}
|
||||
)
|
||||
|
||||
# Remove attributes that are not needed / allowed in the
|
||||
# container recipe
|
||||
for subsection in ('cdash', 'gitlab_ci', 'modules'):
|
||||
if subsection in env_dict:
|
||||
msg = ('the subsection "{0}" in "{1}" is not used when generating'
|
||||
' container recipes and will be discarded')
|
||||
warnings.warn(msg.format(subsection, configuration_file))
|
||||
env_dict.pop(subsection)
|
||||
|
||||
# Set the default value of the concretization strategy to "together" and
|
||||
# warn if the user explicitly set another value
|
||||
env_dict.setdefault('concretization', 'together')
|
||||
if env_dict['concretization'] != 'together':
|
||||
msg = ('the "concretization" attribute of the environment is set '
|
||||
'to "{0}" [the advised value is instead "together"]')
|
||||
warnings.warn(msg.format(env_dict['concretization']))
|
||||
|
||||
# Check if the install tree was explicitly set to a custom value and warn
|
||||
# that it will be overridden
|
||||
environment_config = env_dict.get('config', {})
|
||||
if environment_config.get('install_tree', None):
|
||||
msg = ('the "config:install_tree" attribute has been set explicitly '
|
||||
'and will be overridden in the container image')
|
||||
warnings.warn(msg)
|
||||
|
||||
# Likewise for the view
|
||||
environment_view = env_dict.get('view', None)
|
||||
if environment_view:
|
||||
msg = ('the "view" attribute has been set explicitly '
|
||||
'and will be overridden in the container image')
|
||||
warnings.warn(msg)
|
||||
|
||||
jsonschema.validate(config, schema=env.schema)
|
||||
return config
|
||||
50
lib/spack/spack/container/images.json
Normal file
50
lib/spack/spack/container/images.json
Normal file
@ -0,0 +1,50 @@
|
||||
{
|
||||
"ubuntu:18.04": {
|
||||
"update": "apt-get -yqq update && apt-get -yqq upgrade",
|
||||
"install": "apt-get -yqq install",
|
||||
"clean": "rm -rf /var/lib/apt/lists/*",
|
||||
"environment": [],
|
||||
"build": "spack/ubuntu-bionic",
|
||||
"build_tags": {
|
||||
"develop": "latest",
|
||||
"0.14": "0.14",
|
||||
"0.14.0": "0.14.0"
|
||||
}
|
||||
},
|
||||
"ubuntu:16.04": {
|
||||
"update": "apt-get -yqq update && apt-get -yqq upgrade",
|
||||
"install": "apt-get -yqq install",
|
||||
"clean": "rm -rf /var/lib/apt/lists/*",
|
||||
"environment": [],
|
||||
"build": "spack/ubuntu-xenial",
|
||||
"build_tags": {
|
||||
"develop": "latest",
|
||||
"0.14": "0.14",
|
||||
"0.14.0": "0.14.0"
|
||||
}
|
||||
},
|
||||
"centos:7": {
|
||||
"update": "yum update -y && yum install -y epel-release && yum update -y",
|
||||
"install": "yum install -y",
|
||||
"clean": "rm -rf /var/cache/yum && yum clean all",
|
||||
"environment": [],
|
||||
"build": "spack/centos7",
|
||||
"build_tags": {
|
||||
"develop": "latest",
|
||||
"0.14": "0.14",
|
||||
"0.14.0": "0.14.0"
|
||||
}
|
||||
},
|
||||
"centos:6": {
|
||||
"update": "yum update -y && yum install -y epel-release && yum update -y",
|
||||
"install": "yum install -y",
|
||||
"clean": "rm -rf /var/cache/yum && yum clean all",
|
||||
"environment": [],
|
||||
"build": "spack/centos6",
|
||||
"build_tags": {
|
||||
"develop": "latest",
|
||||
"0.14": "0.14",
|
||||
"0.14.0": "0.14.0"
|
||||
}
|
||||
}
|
||||
}
|
||||
72
lib/spack/spack/container/images.py
Normal file
72
lib/spack/spack/container/images.py
Normal file
@ -0,0 +1,72 @@
|
||||
# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other
|
||||
# Spack Project Developers. See the top-level COPYRIGHT file for details.
|
||||
#
|
||||
# SPDX-License-Identifier: (Apache-2.0 OR MIT)
|
||||
"""Manages the details on the images used in the build and the run stage."""
|
||||
import json
|
||||
import os.path
|
||||
|
||||
#: Global variable used to cache in memory the content of images.json
|
||||
_data = None
|
||||
|
||||
|
||||
def data():
|
||||
"""Returns a dictionary with the static data on the images.
|
||||
|
||||
The dictionary is read from a JSON file lazily the first time
|
||||
this function is called.
|
||||
"""
|
||||
global _data
|
||||
if not _data:
|
||||
json_dir = os.path.abspath(os.path.dirname(__file__))
|
||||
json_file = os.path.join(json_dir, 'images.json')
|
||||
with open(json_file) as f:
|
||||
_data = json.load(f)
|
||||
return _data
|
||||
|
||||
|
||||
def build_info(image, spack_version):
|
||||
"""Returns the name of the build image and its tag.
|
||||
|
||||
Args:
|
||||
image (str): image to be used at run-time. Should be of the form
|
||||
<image_name>:<image_tag> e.g. "ubuntu:18.04"
|
||||
spack_version (str): version of Spack that we want to use to build
|
||||
|
||||
Returns:
|
||||
A tuple with (image_name, image_tag) for the build image
|
||||
"""
|
||||
# Don't handle error here, as a wrong image should have been
|
||||
# caught by the JSON schema
|
||||
image_data = data()[image]
|
||||
build_image = image_data['build']
|
||||
|
||||
# Try to check if we have a tag for this Spack version
|
||||
try:
|
||||
build_tag = image_data['build_tags'][spack_version]
|
||||
except KeyError:
|
||||
msg = ('the image "{0}" has no tag for Spack version "{1}" '
|
||||
'[valid versions are {2}]')
|
||||
msg = msg.format(build_image, spack_version,
|
||||
', '.join(image_data['build_tags'].keys()))
|
||||
raise ValueError(msg)
|
||||
|
||||
return build_image, build_tag
|
||||
|
||||
|
||||
def package_info(image):
|
||||
"""Returns the commands used to update system repositories, install
|
||||
system packages and clean afterwards.
|
||||
|
||||
Args:
|
||||
image (str): image to be used at run-time. Should be of the form
|
||||
<image_name>:<image_tag> e.g. "ubuntu:18.04"
|
||||
|
||||
Returns:
|
||||
A tuple of (update, install, clean) commands.
|
||||
"""
|
||||
image_data = data()[image]
|
||||
update = image_data['update']
|
||||
install = image_data['install']
|
||||
clean = image_data['clean']
|
||||
return update, install, clean
|
||||
154
lib/spack/spack/container/writers/__init__.py
Normal file
154
lib/spack/spack/container/writers/__init__.py
Normal file
@ -0,0 +1,154 @@
|
||||
# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other
|
||||
# Spack Project Developers. See the top-level COPYRIGHT file for details.
|
||||
#
|
||||
# SPDX-License-Identifier: (Apache-2.0 OR MIT)
|
||||
"""Writers for different kind of recipes and related
|
||||
convenience functions.
|
||||
"""
|
||||
import collections
|
||||
import copy
|
||||
|
||||
import spack.environment
|
||||
import spack.schema.env
|
||||
import spack.tengine as tengine
|
||||
import spack.util.spack_yaml as syaml
|
||||
|
||||
from spack.container.images import build_info, package_info
|
||||
|
||||
#: Caches all the writers that are currently supported
|
||||
_writer_factory = {}
|
||||
|
||||
|
||||
def writer(name):
|
||||
"""Decorator to register a factory for a recipe writer.
|
||||
|
||||
Each factory should take a configuration dictionary and return a
|
||||
properly configured writer that, when called, prints the
|
||||
corresponding recipe.
|
||||
"""
|
||||
def _decorator(factory):
|
||||
_writer_factory[name] = factory
|
||||
return factory
|
||||
return _decorator
|
||||
|
||||
|
||||
def create(configuration):
|
||||
"""Returns a writer that conforms to the configuration passed as input.
|
||||
|
||||
Args:
|
||||
configuration: how to generate the current recipe
|
||||
"""
|
||||
name = spack.environment.config_dict(configuration)['container']['format']
|
||||
return _writer_factory[name](configuration)
|
||||
|
||||
|
||||
def recipe(configuration):
|
||||
"""Returns a recipe that conforms to the configuration passed as input.
|
||||
|
||||
Args:
|
||||
configuration: how to generate the current recipe
|
||||
"""
|
||||
return create(configuration)()
|
||||
|
||||
|
||||
class PathContext(tengine.Context):
|
||||
"""Generic context used to instantiate templates of recipes that
|
||||
install software in a common location and make it available
|
||||
directly via PATH.
|
||||
"""
|
||||
def __init__(self, config):
|
||||
self.config = spack.environment.config_dict(config)
|
||||
self.container_config = self.config['container']
|
||||
|
||||
@tengine.context_property
|
||||
def run(self):
|
||||
"""Information related to the run image."""
|
||||
image = self.container_config['base']['image']
|
||||
Run = collections.namedtuple('Run', ['image'])
|
||||
return Run(image=image)
|
||||
|
||||
@tengine.context_property
|
||||
def build(self):
|
||||
"""Information related to the build image."""
|
||||
|
||||
# Map the final image to the correct build image
|
||||
run_image = self.container_config['base']['image']
|
||||
spack_version = self.container_config['base']['spack']
|
||||
image, tag = build_info(run_image, spack_version)
|
||||
|
||||
Build = collections.namedtuple('Build', ['image', 'tag'])
|
||||
return Build(image=image, tag=tag)
|
||||
|
||||
@tengine.context_property
|
||||
def strip(self):
|
||||
"""Whether or not to strip binaries in the image"""
|
||||
return self.container_config.get('strip', True)
|
||||
|
||||
@tengine.context_property
|
||||
def paths(self):
|
||||
"""Important paths in the image"""
|
||||
Paths = collections.namedtuple('Paths', [
|
||||
'environment', 'store', 'view'
|
||||
])
|
||||
return Paths(
|
||||
environment='/opt/spack-environment',
|
||||
store='/opt/software',
|
||||
view='/opt/view'
|
||||
)
|
||||
|
||||
@tengine.context_property
|
||||
def manifest(self):
|
||||
"""The spack.yaml file that should be used in the image"""
|
||||
import jsonschema
|
||||
# Copy in the part of spack.yaml prescribed in the configuration file
|
||||
manifest = copy.deepcopy(self.config)
|
||||
manifest.pop('container')
|
||||
|
||||
# Ensure that a few paths are where they need to be
|
||||
manifest.setdefault('config', syaml.syaml_dict())
|
||||
manifest['config']['install_tree'] = self.paths.store
|
||||
manifest['view'] = self.paths.view
|
||||
manifest = {'spack': manifest}
|
||||
|
||||
# Validate the manifest file
|
||||
jsonschema.validate(manifest, schema=spack.schema.env.schema)
|
||||
|
||||
return syaml.dump(manifest, default_flow_style=False).strip()
|
||||
|
||||
@tengine.context_property
|
||||
def os_packages(self):
|
||||
"""Additional system packages that are needed at run-time."""
|
||||
package_list = self.container_config.get('os_packages', None)
|
||||
if not package_list:
|
||||
return package_list
|
||||
|
||||
image = self.container_config['base']['image']
|
||||
update, install, clean = package_info(image)
|
||||
Packages = collections.namedtuple(
|
||||
'Packages', ['update', 'install', 'list', 'clean']
|
||||
)
|
||||
return Packages(update=update, install=install,
|
||||
list=package_list, clean=clean)
|
||||
|
||||
@tengine.context_property
|
||||
def extra_instructions(self):
|
||||
Extras = collections.namedtuple('Extra', ['build', 'final'])
|
||||
extras = self.container_config.get('extra_instructions', {})
|
||||
build, final = extras.get('build', None), extras.get('final', None)
|
||||
return Extras(build=build, final=final)
|
||||
|
||||
@tengine.context_property
|
||||
def labels(self):
|
||||
return self.container_config.get('labels', {})
|
||||
|
||||
def __call__(self):
|
||||
"""Returns the recipe as a string"""
|
||||
env = tengine.make_environment()
|
||||
t = env.get_template(self.template_name)
|
||||
return t.render(**self.to_dict())
|
||||
|
||||
|
||||
# Import after function definition all the modules in this package,
|
||||
# so that registration of writers will happen automatically
|
||||
import spack.container.writers.singularity # noqa
|
||||
import spack.container.writers.docker # noqa
|
||||
30
lib/spack/spack/container/writers/docker.py
Normal file
30
lib/spack/spack/container/writers/docker.py
Normal file
@ -0,0 +1,30 @@
|
||||
# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other
|
||||
# Spack Project Developers. See the top-level COPYRIGHT file for details.
|
||||
#
|
||||
# SPDX-License-Identifier: (Apache-2.0 OR MIT)
|
||||
import spack.tengine as tengine
|
||||
|
||||
from . import writer, PathContext
|
||||
|
||||
|
||||
@writer('docker')
|
||||
class DockerContext(PathContext):
|
||||
"""Context used to instantiate a Dockerfile"""
|
||||
#: Name of the template used for Dockerfiles
|
||||
template_name = 'container/Dockerfile'
|
||||
|
||||
@tengine.context_property
|
||||
def manifest(self):
|
||||
manifest_str = super(DockerContext, self).manifest
|
||||
# Docker doesn't support HEREDOC so we need to resort to
|
||||
# a horrible echo trick to have the manifest in the Dockerfile
|
||||
echoed_lines = []
|
||||
for idx, line in enumerate(manifest_str.split('\n')):
|
||||
if idx == 0:
|
||||
echoed_lines.append('&& (echo "' + line + '" \\')
|
||||
continue
|
||||
echoed_lines.append('&& echo "' + line + '" \\')
|
||||
|
||||
echoed_lines[-1] = echoed_lines[-1].replace(' \\', ')')
|
||||
|
||||
return '\n'.join(echoed_lines)
|
||||
33
lib/spack/spack/container/writers/singularity.py
Normal file
33
lib/spack/spack/container/writers/singularity.py
Normal file
@ -0,0 +1,33 @@
|
||||
# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other
|
||||
# Spack Project Developers. See the top-level COPYRIGHT file for details.
|
||||
#
|
||||
# SPDX-License-Identifier: (Apache-2.0 OR MIT)
|
||||
import spack.tengine as tengine
|
||||
from . import writer, PathContext
|
||||
|
||||
|
||||
@writer('singularity')
|
||||
class SingularityContext(PathContext):
|
||||
"""Context used to instantiate a Singularity definition file"""
|
||||
#: Name of the template used for Singularity definition files
|
||||
template_name = 'container/singularity.def'
|
||||
|
||||
@property
|
||||
def singularity_config(self):
|
||||
return self.container_config.get('singularity', {})
|
||||
|
||||
@tengine.context_property
|
||||
def runscript(self):
|
||||
return self.singularity_config.get('runscript', '')
|
||||
|
||||
@tengine.context_property
|
||||
def startscript(self):
|
||||
return self.singularity_config.get('startscript', '')
|
||||
|
||||
@tengine.context_property
|
||||
def test(self):
|
||||
return self.singularity_config.get('test', '')
|
||||
|
||||
@tengine.context_property
|
||||
def help(self):
|
||||
return self.singularity_config.get('help', '')
|
||||
82
lib/spack/spack/schema/container.py
Normal file
82
lib/spack/spack/schema/container.py
Normal file
@ -0,0 +1,82 @@
|
||||
# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other
|
||||
# Spack Project Developers. See the top-level COPYRIGHT file for details.
|
||||
#
|
||||
# SPDX-License-Identifier: (Apache-2.0 OR MIT)
|
||||
"""Schema for the 'container' subsection of Spack environments."""
|
||||
|
||||
#: Schema for the container attribute included in Spack environments
|
||||
container_schema = {
|
||||
'type': 'object',
|
||||
'additionalProperties': False,
|
||||
'properties': {
|
||||
# The recipe formats that are currently supported by the command
|
||||
'format': {
|
||||
'type': 'string',
|
||||
'enum': ['docker', 'singularity']
|
||||
},
|
||||
# Describes the base image to start from and the version
|
||||
# of Spack to be used
|
||||
'base': {
|
||||
'type': 'object',
|
||||
'additionalProperties': False,
|
||||
'properties': {
|
||||
'image': {
|
||||
'type': 'string',
|
||||
'enum': ['ubuntu:18.04',
|
||||
'ubuntu:16.04',
|
||||
'centos:7',
|
||||
'centos:6']
|
||||
},
|
||||
'spack': {
|
||||
'type': 'string',
|
||||
'enum': ['develop', '0.14', '0.14.0']
|
||||
}
|
||||
},
|
||||
'required': ['image', 'spack']
|
||||
},
|
||||
# Whether or not to strip installed binaries
|
||||
'strip': {
|
||||
'type': 'boolean',
|
||||
'default': True
|
||||
},
|
||||
# Additional system packages that are needed at runtime
|
||||
'os_packages': {
|
||||
'type': 'array',
|
||||
'items': {
|
||||
'type': 'string'
|
||||
}
|
||||
},
|
||||
# Add labels to the image
|
||||
'labels': {
|
||||
'type': 'object',
|
||||
},
|
||||
# Add a custom extra section at the bottom of a stage
|
||||
'extra_instructions': {
|
||||
'type': 'object',
|
||||
'additionalProperties': False,
|
||||
'properties': {
|
||||
'build': {'type': 'string'},
|
||||
'final': {'type': 'string'}
|
||||
}
|
||||
},
|
||||
# Reserved for properties that are specific to each format
|
||||
'singularity': {
|
||||
'type': 'object',
|
||||
'additionalProperties': False,
|
||||
'default': {},
|
||||
'properties': {
|
||||
'runscript': {'type': 'string'},
|
||||
'startscript': {'type': 'string'},
|
||||
'test': {'type': 'string'},
|
||||
'help': {'type': 'string'}
|
||||
}
|
||||
},
|
||||
'docker': {
|
||||
'type': 'object',
|
||||
'additionalProperties': False,
|
||||
'default': {},
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
properties = {'container': container_schema}
|
||||
@ -13,6 +13,7 @@
|
||||
import spack.schema.cdash
|
||||
import spack.schema.compilers
|
||||
import spack.schema.config
|
||||
import spack.schema.container
|
||||
import spack.schema.gitlab_ci
|
||||
import spack.schema.mirrors
|
||||
import spack.schema.modules
|
||||
@ -26,6 +27,7 @@
|
||||
spack.schema.cdash.properties,
|
||||
spack.schema.compilers.properties,
|
||||
spack.schema.config.properties,
|
||||
spack.schema.container.properties,
|
||||
spack.schema.gitlab_ci.properties,
|
||||
spack.schema.mirrors.properties,
|
||||
spack.schema.modules.properties,
|
||||
|
||||
@ -30,7 +30,9 @@ def test_packages_are_removed(config, mutable_database, capsys):
|
||||
|
||||
|
||||
@pytest.mark.db
|
||||
def test_gc_with_environment(config, mutable_database, capsys):
|
||||
def test_gc_with_environment(
|
||||
config, mutable_database, mutable_mock_env_path, capsys
|
||||
):
|
||||
s = spack.spec.Spec('simple-inheritance')
|
||||
s.concretize()
|
||||
s.package.do_install(fake=True, explicit=True)
|
||||
|
||||
@ -1,4 +1,4 @@
|
||||
# Copyright 2013-2019 Lawrence Livermore National Security, LLC and other
|
||||
# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other
|
||||
# Spack Project Developers. See the top-level COPYRIGHT file for details.
|
||||
#
|
||||
# SPDX-License-Identifier: (Apache-2.0 OR MIT)
|
||||
|
||||
16
lib/spack/spack/test/container/cli.py
Normal file
16
lib/spack/spack/test/container/cli.py
Normal file
@ -0,0 +1,16 @@
|
||||
# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other
|
||||
# Spack Project Developers. See the top-level COPYRIGHT file for details.
|
||||
#
|
||||
# SPDX-License-Identifier: (Apache-2.0 OR MIT)
|
||||
import llnl.util.filesystem as fs
|
||||
import spack.main
|
||||
|
||||
|
||||
containerize = spack.main.SpackCommand('containerize')
|
||||
|
||||
|
||||
def test_command(configuration_dir, capsys):
|
||||
with capsys.disabled():
|
||||
with fs.working_dir(configuration_dir):
|
||||
output = containerize()
|
||||
assert 'FROM spack/ubuntu-bionic' in output
|
||||
43
lib/spack/spack/test/container/conftest.py
Normal file
43
lib/spack/spack/test/container/conftest.py
Normal file
@ -0,0 +1,43 @@
|
||||
# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other
|
||||
# Spack Project Developers. See the top-level COPYRIGHT file for details.
|
||||
#
|
||||
# SPDX-License-Identifier: (Apache-2.0 OR MIT)
|
||||
import pytest
|
||||
|
||||
import spack.util.spack_yaml as syaml
|
||||
|
||||
|
||||
@pytest.fixture()
|
||||
def minimal_configuration():
|
||||
return {
|
||||
'spack': {
|
||||
'specs': [
|
||||
'gromacs',
|
||||
'mpich',
|
||||
'fftw precision=float'
|
||||
],
|
||||
'container': {
|
||||
'format': 'docker',
|
||||
'base': {
|
||||
'image': 'ubuntu:18.04',
|
||||
'spack': 'develop'
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@pytest.fixture()
|
||||
def config_dumper(tmpdir):
|
||||
"""Function that dumps an environment config in a temporary folder."""
|
||||
def dumper(configuration):
|
||||
content = syaml.dump(configuration, default_flow_style=False)
|
||||
config_file = tmpdir / 'spack.yaml'
|
||||
config_file.write(content)
|
||||
return str(tmpdir)
|
||||
return dumper
|
||||
|
||||
|
||||
@pytest.fixture()
|
||||
def configuration_dir(minimal_configuration, config_dumper):
|
||||
return config_dumper(minimal_configuration)
|
||||
74
lib/spack/spack/test/container/docker.py
Normal file
74
lib/spack/spack/test/container/docker.py
Normal file
@ -0,0 +1,74 @@
|
||||
# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other
|
||||
# Spack Project Developers. See the top-level COPYRIGHT file for details.
|
||||
#
|
||||
# SPDX-License-Identifier: (Apache-2.0 OR MIT)
|
||||
|
||||
import spack.container.writers as writers
|
||||
|
||||
|
||||
def test_manifest(minimal_configuration):
|
||||
writer = writers.create(minimal_configuration)
|
||||
manifest_str = writer.manifest
|
||||
for line in manifest_str.split('\n'):
|
||||
assert 'echo' in line
|
||||
|
||||
|
||||
def test_build_and_run_images(minimal_configuration):
|
||||
writer = writers.create(minimal_configuration)
|
||||
|
||||
# Test the output of run property
|
||||
run = writer.run
|
||||
assert run.image == 'ubuntu:18.04'
|
||||
|
||||
# Test the output of the build property
|
||||
build = writer.build
|
||||
assert build.image == 'spack/ubuntu-bionic'
|
||||
assert build.tag == 'latest'
|
||||
|
||||
|
||||
def test_packages(minimal_configuration):
|
||||
# In this minimal configuration we don't have packages
|
||||
writer = writers.create(minimal_configuration)
|
||||
assert writer.os_packages is None
|
||||
|
||||
# If we add them a list should be returned
|
||||
pkgs = ['libgomp1']
|
||||
minimal_configuration['spack']['container']['os_packages'] = pkgs
|
||||
writer = writers.create(minimal_configuration)
|
||||
p = writer.os_packages
|
||||
assert p.update
|
||||
assert p.install
|
||||
assert p.clean
|
||||
assert p.list == pkgs
|
||||
|
||||
|
||||
def test_ensure_render_works(minimal_configuration):
|
||||
# Here we just want to ensure that nothing is raised
|
||||
writer = writers.create(minimal_configuration)
|
||||
writer()
|
||||
|
||||
|
||||
def test_strip_is_set_from_config(minimal_configuration):
|
||||
writer = writers.create(minimal_configuration)
|
||||
assert writer.strip is True
|
||||
|
||||
minimal_configuration['spack']['container']['strip'] = False
|
||||
writer = writers.create(minimal_configuration)
|
||||
assert writer.strip is False
|
||||
|
||||
|
||||
def test_extra_instructions_is_set_from_config(minimal_configuration):
|
||||
writer = writers.create(minimal_configuration)
|
||||
assert writer.extra_instructions == (None, None)
|
||||
|
||||
test_line = 'RUN echo Hello world!'
|
||||
e = minimal_configuration['spack']['container']
|
||||
e['extra_instructions'] = {}
|
||||
e['extra_instructions']['build'] = test_line
|
||||
writer = writers.create(minimal_configuration)
|
||||
assert writer.extra_instructions == (test_line, None)
|
||||
|
||||
e['extra_instructions']['final'] = test_line
|
||||
del e['extra_instructions']['build']
|
||||
writer = writers.create(minimal_configuration)
|
||||
assert writer.extra_instructions == (None, test_line)
|
||||
58
lib/spack/spack/test/container/images.py
Normal file
58
lib/spack/spack/test/container/images.py
Normal file
@ -0,0 +1,58 @@
|
||||
# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other
|
||||
# Spack Project Developers. See the top-level COPYRIGHT file for details.
|
||||
#
|
||||
# SPDX-License-Identifier: (Apache-2.0 OR MIT)
|
||||
import os.path
|
||||
|
||||
import pytest
|
||||
|
||||
import spack.container
|
||||
|
||||
|
||||
@pytest.mark.parametrize('image,spack_version,expected', [
|
||||
('ubuntu:18.04', 'develop', ('spack/ubuntu-bionic', 'latest')),
|
||||
('ubuntu:18.04', '0.14.0', ('spack/ubuntu-bionic', '0.14.0')),
|
||||
])
|
||||
def test_build_info(image, spack_version, expected):
|
||||
output = spack.container.images.build_info(image, spack_version)
|
||||
assert output == expected
|
||||
|
||||
|
||||
@pytest.mark.parametrize('image,spack_version', [
|
||||
('ubuntu:18.04', 'doesnotexist')
|
||||
])
|
||||
def test_build_info_error(image, spack_version):
|
||||
with pytest.raises(ValueError, match=r"has no tag for"):
|
||||
spack.container.images.build_info(image, spack_version)
|
||||
|
||||
|
||||
@pytest.mark.parametrize('image', [
|
||||
'ubuntu:18.04'
|
||||
])
|
||||
def test_package_info(image):
|
||||
update, install, clean = spack.container.images.package_info(image)
|
||||
assert update
|
||||
assert install
|
||||
assert clean
|
||||
|
||||
|
||||
@pytest.mark.parametrize('extra_config,expected_msg', [
|
||||
({'modules': {'enable': ['tcl']}}, 'the subsection "modules" in'),
|
||||
({'concretization': 'separately'}, 'the "concretization" attribute'),
|
||||
({'config': {'install_tree': '/some/dir'}},
|
||||
'the "config:install_tree" attribute has been set'),
|
||||
({'view': '/some/dir'}, 'the "view" attribute has been set')
|
||||
])
|
||||
def test_validate(
|
||||
extra_config, expected_msg, minimal_configuration, config_dumper
|
||||
):
|
||||
minimal_configuration['spack'].update(extra_config)
|
||||
spack_yaml_dir = config_dumper(minimal_configuration)
|
||||
spack_yaml = os.path.join(spack_yaml_dir, 'spack.yaml')
|
||||
|
||||
with pytest.warns(UserWarning) as w:
|
||||
spack.container.validate(spack_yaml)
|
||||
|
||||
# Tests are designed to raise only one warning
|
||||
assert len(w) == 1
|
||||
assert expected_msg in str(w.pop().message)
|
||||
16
lib/spack/spack/test/container/schema.py
Normal file
16
lib/spack/spack/test/container/schema.py
Normal file
@ -0,0 +1,16 @@
|
||||
# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other
|
||||
# Spack Project Developers. See the top-level COPYRIGHT file for details.
|
||||
#
|
||||
# SPDX-License-Identifier: (Apache-2.0 OR MIT)
|
||||
|
||||
import spack.container
|
||||
import spack.schema.container
|
||||
|
||||
|
||||
def test_images_in_schema():
|
||||
properties = spack.schema.container.container_schema['properties']
|
||||
allowed_images = set(
|
||||
properties['base']['properties']['image']['enum']
|
||||
)
|
||||
images_in_json = set(x for x in spack.container.images.data())
|
||||
assert images_in_json == allowed_images
|
||||
42
lib/spack/spack/test/container/singularity.py
Normal file
42
lib/spack/spack/test/container/singularity.py
Normal file
@ -0,0 +1,42 @@
|
||||
# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other
|
||||
# Spack Project Developers. See the top-level COPYRIGHT file for details.
|
||||
#
|
||||
# SPDX-License-Identifier: (Apache-2.0 OR MIT)
|
||||
import pytest
|
||||
|
||||
import spack.container.writers as writers
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def singularity_configuration(minimal_configuration):
|
||||
minimal_configuration['spack']['container']['format'] = 'singularity'
|
||||
return minimal_configuration
|
||||
|
||||
|
||||
def test_ensure_render_works(singularity_configuration):
|
||||
container_config = singularity_configuration['spack']['container']
|
||||
assert container_config['format'] == 'singularity'
|
||||
# Here we just want to ensure that nothing is raised
|
||||
writer = writers.create(singularity_configuration)
|
||||
writer()
|
||||
|
||||
|
||||
@pytest.mark.parametrize('properties,expected', [
|
||||
({'runscript': '/opt/view/bin/h5ls'},
|
||||
{'runscript': '/opt/view/bin/h5ls',
|
||||
'startscript': '',
|
||||
'test': '',
|
||||
'help': ''})
|
||||
])
|
||||
def test_singularity_specific_properties(
|
||||
properties, expected, singularity_configuration
|
||||
):
|
||||
# Set the property in the configuration
|
||||
container_config = singularity_configuration['spack']['container']
|
||||
for name, value in properties.items():
|
||||
container_config.setdefault('singularity', {})[name] = value
|
||||
|
||||
# Assert the properties return the expected values
|
||||
writer = writers.create(singularity_configuration)
|
||||
for name, value in expected.items():
|
||||
assert getattr(writer, name) == value
|
||||
@ -313,7 +313,7 @@ _spack() {
|
||||
then
|
||||
SPACK_COMPREPLY="-h --help -H --all-help --color -C --config-scope -d --debug --timestamp --pdb -e --env -D --env-dir -E --no-env --use-env-repo -k --insecure -l --enable-locks -L --disable-locks -m --mock -p --profile --sorted-profile --lines -v --verbose --stacktrace -V --version --print-shell-vars"
|
||||
else
|
||||
SPACK_COMPREPLY="activate add arch blame bootstrap build build-env buildcache cd checksum ci clean clone commands compiler compilers concretize config configure create deactivate debug dependencies dependents deprecate dev-build diy docs edit env extensions fetch find flake8 gc gpg graph help info install license list load location log-parse maintainers mirror module patch pkg providers pydoc python reindex remove rm repo resource restage setup spec stage test uninstall unload upload-s3 url verify versions view"
|
||||
SPACK_COMPREPLY="activate add arch blame bootstrap build build-env buildcache cd checksum ci clean clone commands compiler compilers concretize config configure containerize create deactivate debug dependencies dependents deprecate dev-build diy docs edit env extensions fetch find flake8 gc gpg graph help info install license list load location log-parse maintainers mirror module patch pkg providers pydoc python reindex remove rm repo resource restage setup spec stage test uninstall unload upload-s3 url verify versions view"
|
||||
fi
|
||||
}
|
||||
|
||||
@ -628,6 +628,10 @@ _spack_configure() {
|
||||
fi
|
||||
}
|
||||
|
||||
_spack_containerize() {
|
||||
SPACK_COMPREPLY="-h --help"
|
||||
}
|
||||
|
||||
_spack_create() {
|
||||
if $list_options
|
||||
then
|
||||
|
||||
51
share/spack/templates/container/Dockerfile
Normal file
51
share/spack/templates/container/Dockerfile
Normal file
@ -0,0 +1,51 @@
|
||||
# Build stage with Spack pre-installed and ready to be used
|
||||
FROM {{ build.image }}:{{ build.tag }} as builder
|
||||
|
||||
# What we want to install and how we want to install it
|
||||
# is specified in a manifest file (spack.yaml)
|
||||
RUN mkdir {{ paths.environment }} \
|
||||
{{ manifest }} > {{ paths.environment }}/spack.yaml
|
||||
|
||||
# Install the software, remove unecessary deps
|
||||
RUN cd {{ paths.environment }} && spack install && spack gc -y
|
||||
{% if strip %}
|
||||
|
||||
# Strip all the binaries
|
||||
RUN find -L {{ paths.view }}/* -type f -exec readlink -f '{}' \; | \
|
||||
xargs file -i | \
|
||||
grep 'charset=binary' | \
|
||||
grep 'x-executable\|x-archive\|x-sharedlib' | \
|
||||
awk -F: '{print $1}' | xargs strip -s
|
||||
{% endif %}
|
||||
|
||||
# Modifications to the environment that are necessary to run
|
||||
RUN cd {{ paths.environment }} && \
|
||||
spack env activate --sh -d . >> /etc/profile.d/z10_spack_environment.sh
|
||||
|
||||
{% if extra_instructions.build %}
|
||||
{{ extra_instructions.build }}
|
||||
{% endif %}
|
||||
|
||||
# Bare OS image to run the installed executables
|
||||
FROM {{ run.image }}
|
||||
|
||||
COPY --from=builder {{ paths.environment }} {{ paths.environment }}
|
||||
COPY --from=builder {{ paths.store }} {{ paths.store }}
|
||||
COPY --from=builder {{ paths.view }} {{ paths.view }}
|
||||
COPY --from=builder /etc/profile.d/z10_spack_environment.sh /etc/profile.d/z10_spack_environment.sh
|
||||
|
||||
{% if os_packages %}
|
||||
RUN {{ os_packages.update }} \
|
||||
&& {{ os_packages.install }}{% for pkg in os_packages.list %} {{ pkg }}{% endfor %} \
|
||||
&& {{ os_packages.clean }}
|
||||
{% endif %}
|
||||
|
||||
{% if extra_instructions.final %}
|
||||
{{ extra_instructions.final }}
|
||||
{% endif %}
|
||||
|
||||
{% for label, value in labels.items() %}
|
||||
LABEL "{{ label }}"="{{ value }}"
|
||||
{% endfor %}
|
||||
|
||||
ENTRYPOINT ["/bin/bash", "--rcfile", "/etc/profile", "-l"]
|
||||
90
share/spack/templates/container/singularity.def
Normal file
90
share/spack/templates/container/singularity.def
Normal file
@ -0,0 +1,90 @@
|
||||
Bootstrap: docker
|
||||
From: {{ build.image }}:{{ build.tag }}
|
||||
Stage: build
|
||||
|
||||
%post
|
||||
# Create the manifest file for the installation in /opt/spack-environment
|
||||
mkdir {{ paths.environment }} && cd {{ paths.environment }}
|
||||
cat << EOF > spack.yaml
|
||||
{{ manifest }}
|
||||
EOF
|
||||
|
||||
# Install all the required software
|
||||
. /opt/spack/share/spack/setup-env.sh
|
||||
spack install
|
||||
spack gc -y
|
||||
spack env activate --sh -d . >> {{ paths.environment }}/environment_modifications.sh
|
||||
{% if strip %}
|
||||
|
||||
# Strip the binaries to reduce the size of the image
|
||||
find -L {{ paths.view }}/* -type f -exec readlink -f '{}' \; | \
|
||||
xargs file -i | \
|
||||
grep 'charset=binary' | \
|
||||
grep 'x-executable\|x-archive\|x-sharedlib' | \
|
||||
awk -F: '{print $1}' | xargs strip -s
|
||||
{% endif %}
|
||||
{% if extra_instructions.build %}
|
||||
{{ extra_instructions.build }}
|
||||
{% endif %}
|
||||
|
||||
|
||||
{% if apps %}
|
||||
{% for application, help_text in apps.items() %}
|
||||
|
||||
%apprun {{ application }}
|
||||
exec /opt/view/bin/{{ application }} "$@"
|
||||
|
||||
%apphelp {{ application }}
|
||||
{{help_text }}
|
||||
{% endfor %}
|
||||
{% endif %}
|
||||
|
||||
Bootstrap: docker
|
||||
From: {{ run.image }}
|
||||
Stage: final
|
||||
|
||||
%files from build
|
||||
{{ paths.environment }} /opt
|
||||
{{ paths.store }} /opt
|
||||
{{ paths.view }} /opt
|
||||
{{ paths.environment }}/environment_modifications.sh {{ paths.environment }}/environment_modifications.sh
|
||||
|
||||
%post
|
||||
{% if os_packages.list %}
|
||||
# Update, install and cleanup of system packages
|
||||
{{ os_packages.update }}
|
||||
{{ os_packages.install }} {{ os_packages.list | join | replace('\n', ' ') }}
|
||||
{{ os_packages.clean }}
|
||||
{% endif %}
|
||||
# Modify the environment without relying on sourcing shell specific files at startup
|
||||
cat {{ paths.environment }}/environment_modifications.sh >> $SINGULARITY_ENVIRONMENT
|
||||
{% if extra_instructions.final %}
|
||||
{{ extra_instructions.final }}
|
||||
{% endif %}
|
||||
|
||||
{% if runscript %}
|
||||
%runscript
|
||||
{{ runscript }}
|
||||
{% endif %}
|
||||
|
||||
{% if startscript %}
|
||||
%startscript
|
||||
{{ startscript }}
|
||||
{% endif %}
|
||||
|
||||
{% if test %}
|
||||
%test
|
||||
{{ test }}
|
||||
{% endif %}
|
||||
|
||||
{% if help %}
|
||||
%help
|
||||
{{ help }}
|
||||
{% endif %}
|
||||
|
||||
{% if labels %}
|
||||
%labels
|
||||
{% for label, value in labels.items() %}
|
||||
{{ label }} {{ value }}
|
||||
{% endfor %}
|
||||
{% endif %}
|
||||
Loading…
Reference in New Issue
Block a user