spack containerize generates containers from envs (#14202)
This PR adds a new command to Spack:
```console
$ spack containerize -h
usage: spack containerize [-h] [--config CONFIG]
creates recipes to build images for different container runtimes
optional arguments:
-h, --help show this help message and exit
--config CONFIG configuration for the container recipe that will be generated
```
which takes an environment with an additional `container` section:
```yaml
spack:
specs:
- gromacs build_type=Release
- mpich
- fftw precision=float
packages:
all:
target: [broadwell]
container:
# Select the format of the recipe e.g. docker,
# singularity or anything else that is currently supported
format: docker
# Select from a valid list of images
base:
image: "ubuntu:18.04"
spack: prerelease
# Additional system packages that are needed at runtime
os_packages:
- libgomp1
```
and turns it into a `Dockerfile` or a Singularity definition file, for instance:
```Dockerfile
# Build stage with Spack pre-installed and ready to be used
FROM spack/ubuntu-bionic:prerelease as builder
# What we want to install and how we want to install it
# is specified in a manifest file (spack.yaml)
RUN mkdir /opt/spack-environment \
&& (echo "spack:" \
&& echo " specs:" \
&& echo " - gromacs build_type=Release" \
&& echo " - mpich" \
&& echo " - fftw precision=float" \
&& echo " packages:" \
&& echo " all:" \
&& echo " target:" \
&& echo " - broadwell" \
&& echo " config:" \
&& echo " install_tree: /opt/software" \
&& echo " concretization: together" \
&& echo " view: /opt/view") > /opt/spack-environment/spack.yaml
# Install the software, remove unecessary deps and strip executables
RUN cd /opt/spack-environment && spack install && spack autoremove -y
RUN find -L /opt/view/* -type f -exec readlink -f '{}' \; | \
xargs file -i | \
grep 'charset=binary' | \
grep 'x-executable\|x-archive\|x-sharedlib' | \
awk -F: '{print $1}' | xargs strip -s
# Modifications to the environment that are necessary to run
RUN cd /opt/spack-environment && \
spack env activate --sh -d . >> /etc/profile.d/z10_spack_environment.sh
# Bare OS image to run the installed executables
FROM ubuntu:18.04
COPY --from=builder /opt/spack-environment /opt/spack-environment
COPY --from=builder /opt/software /opt/software
COPY --from=builder /opt/view /opt/view
COPY --from=builder /etc/profile.d/z10_spack_environment.sh /etc/profile.d/z10_spack_environment.sh
RUN apt-get -yqq update && apt-get -yqq upgrade \
&& apt-get -yqq install libgomp1 \
&& rm -rf /var/lib/apt/lists/*
ENTRYPOINT ["/bin/bash", "--rcfile", "/etc/profile", "-l"]
```
This commit is contained in:
Massimiliano Culpo
committed by
GitHub
GitHub
parent
ed501eaab2
commit
9635ff3d20
307
lib/spack/docs/containers.rst
Normal file
307
lib/spack/docs/containers.rst
Normal file
@@ -0,0 +1,307 @@
|
||||
.. Copyright 2013-2020 Lawrence Livermore National Security, LLC and other
|
||||
Spack Project Developers. See the top-level COPYRIGHT file for details.
|
||||
|
||||
SPDX-License-Identifier: (Apache-2.0 OR MIT)
|
||||
|
||||
.. _containers:
|
||||
|
||||
================
|
||||
Container Images
|
||||
================
|
||||
|
||||
Spack can be an ideal tool to setup images for containers since all the
|
||||
features discussed in :ref:`environments` can greatly help to manage
|
||||
the installation of software during the image build process. Nonetheless,
|
||||
building a production image from scratch still requires a lot of
|
||||
boilerplate to:
|
||||
|
||||
- Get Spack working within the image, possibly running as root
|
||||
- Minimize the physical size of the software installed
|
||||
- Properly update the system software in the base image
|
||||
|
||||
To facilitate users with these tedious tasks, Spack provides a command
|
||||
to automatically generate recipes for container images based on
|
||||
Environments:
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
$ ls
|
||||
spack.yaml
|
||||
|
||||
$ spack containerize
|
||||
# Build stage with Spack pre-installed and ready to be used
|
||||
FROM spack/centos7:latest as builder
|
||||
|
||||
# What we want to install and how we want to install it
|
||||
# is specified in a manifest file (spack.yaml)
|
||||
RUN mkdir /opt/spack-environment \
|
||||
&& (echo "spack:" \
|
||||
&& echo " specs:" \
|
||||
&& echo " - gromacs+mpi" \
|
||||
&& echo " - mpich" \
|
||||
&& echo " concretization: together" \
|
||||
&& echo " config:" \
|
||||
&& echo " install_tree: /opt/software" \
|
||||
&& echo " view: /opt/view") > /opt/spack-environment/spack.yaml
|
||||
|
||||
# Install the software, remove unecessary deps
|
||||
RUN cd /opt/spack-environment && spack install && spack gc -y
|
||||
|
||||
# Strip all the binaries
|
||||
RUN find -L /opt/view/* -type f -exec readlink -f '{}' \; | \
|
||||
xargs file -i | \
|
||||
grep 'charset=binary' | \
|
||||
grep 'x-executable\|x-archive\|x-sharedlib' | \
|
||||
awk -F: '{print $1}' | xargs strip -s
|
||||
|
||||
# Modifications to the environment that are necessary to run
|
||||
RUN cd /opt/spack-environment && \
|
||||
spack env activate --sh -d . >> /etc/profile.d/z10_spack_environment.sh
|
||||
|
||||
|
||||
# Bare OS image to run the installed executables
|
||||
FROM centos:7
|
||||
|
||||
COPY --from=builder /opt/spack-environment /opt/spack-environment
|
||||
COPY --from=builder /opt/software /opt/software
|
||||
COPY --from=builder /opt/view /opt/view
|
||||
COPY --from=builder /etc/profile.d/z10_spack_environment.sh /etc/profile.d/z10_spack_environment.sh
|
||||
|
||||
RUN yum update -y && yum install -y epel-release && yum update -y \
|
||||
&& yum install -y libgomp \
|
||||
&& rm -rf /var/cache/yum && yum clean all
|
||||
|
||||
RUN echo 'export PS1="\[$(tput bold)\]\[$(tput setaf 1)\][gromacs]\[$(tput setaf 2)\]\u\[$(tput sgr0)\]:\w $ \[$(tput sgr0)\]"' >> ~/.bashrc
|
||||
|
||||
|
||||
LABEL "app"="gromacs"
|
||||
LABEL "mpi"="mpich"
|
||||
|
||||
ENTRYPOINT ["/bin/bash", "--rcfile", "/etc/profile", "-l"]
|
||||
|
||||
|
||||
The bits that make this automation possible are discussed in details
|
||||
below. All the images generated in this way will be based on
|
||||
multi-stage builds with:
|
||||
|
||||
- A fat ``build`` stage containing common build tools and Spack itself
|
||||
- A minimal ``final`` stage containing only the software requested by the user
|
||||
|
||||
-----------------
|
||||
Spack Base Images
|
||||
-----------------
|
||||
|
||||
Docker images with Spack preinstalled and ready to be used are
|
||||
built on `Docker Hub <https://hub.docker.com/u/spack>`_
|
||||
at every push to ``develop`` or to a release branch. The OS that
|
||||
are currently supported are summarized in the table below:
|
||||
|
||||
.. _containers-supported-os:
|
||||
|
||||
.. list-table:: Supported operating systems
|
||||
:header-rows: 1
|
||||
|
||||
* - Operating System
|
||||
- Base Image
|
||||
- Spack Image
|
||||
* - Ubuntu 16.04
|
||||
- ``ubuntu:16.04``
|
||||
- ``spack/ubuntu-xenial``
|
||||
* - Ubuntu 18.04
|
||||
- ``ubuntu:16.04``
|
||||
- ``spack/ubuntu-bionic``
|
||||
* - CentOS 6
|
||||
- ``centos:6``
|
||||
- ``spack/centos6``
|
||||
* - CentOS 7
|
||||
- ``centos:7``
|
||||
- ``spack/centos7``
|
||||
|
||||
All the images are tagged with the corresponding release of Spack:
|
||||
|
||||
.. image:: dockerhub_spack.png
|
||||
|
||||
with the exception of the ``latest`` tag that points to the HEAD
|
||||
of the ``develop`` branch. These images are available for anyone
|
||||
to use and take care of all the repetitive tasks that are necessary
|
||||
to setup Spack within a container. All the container recipes generated
|
||||
automatically by Spack use them as base images for their ``build`` stage.
|
||||
|
||||
|
||||
-------------------------
|
||||
Environment Configuration
|
||||
-------------------------
|
||||
|
||||
Any Spack Environment can be used for the automatic generation of container
|
||||
recipes. Sensible defaults are provided for things like the base image or the
|
||||
version of Spack used in the image. If a finer tuning is needed it can be
|
||||
obtained by adding the relevant metadata under the ``container`` attribute
|
||||
of environments:
|
||||
|
||||
.. code-block:: yaml
|
||||
|
||||
spack:
|
||||
specs:
|
||||
- gromacs+mpi
|
||||
- mpich
|
||||
|
||||
container:
|
||||
# Select the format of the recipe e.g. docker,
|
||||
# singularity or anything else that is currently supported
|
||||
format: docker
|
||||
|
||||
# Select from a valid list of images
|
||||
base:
|
||||
image: "centos:7"
|
||||
spack: develop
|
||||
|
||||
# Whether or not to strip binaries
|
||||
strip: true
|
||||
|
||||
# Additional system packages that are needed at runtime
|
||||
os_packages:
|
||||
- libgomp
|
||||
|
||||
# Extra instructions
|
||||
extra_instructions:
|
||||
final: |
|
||||
RUN echo 'export PS1="\[$(tput bold)\]\[$(tput setaf 1)\][gromacs]\[$(tput setaf 2)\]\u\[$(tput sgr0)\]:\w $ \[$(tput sgr0)\]"' >> ~/.bashrc
|
||||
|
||||
# Labels for the image
|
||||
labels:
|
||||
app: "gromacs"
|
||||
mpi: "mpich"
|
||||
|
||||
The tables below describe the configuration options that are currently supported:
|
||||
|
||||
.. list-table:: General configuration options for the ``container`` section of ``spack.yaml``
|
||||
:header-rows: 1
|
||||
|
||||
* - Option Name
|
||||
- Description
|
||||
- Allowed Values
|
||||
- Required
|
||||
* - ``format``
|
||||
- The format of the recipe
|
||||
- ``docker`` or ``singularity``
|
||||
- Yes
|
||||
* - ``base:image``
|
||||
- Base image for ``final`` stage
|
||||
- See :ref:`containers-supported-os`
|
||||
- Yes
|
||||
* - ``base:spack``
|
||||
- Version of Spack
|
||||
- Valid tags for ``base:image``
|
||||
- Yes
|
||||
* - ``strip``
|
||||
- Whether to strip binaries
|
||||
- ``true`` (default) or ``false``
|
||||
- No
|
||||
* - ``os_packages``
|
||||
- System packages to be installed
|
||||
- Valid packages for the ``final`` OS
|
||||
- No
|
||||
* - ``extra_instructions:build``
|
||||
- Extra instructions (e.g. `RUN`, `COPY`, etc.) at the end of the ``build`` stage
|
||||
- Anything understood by the current ``format``
|
||||
- No
|
||||
* - ``extra_instructions:final``
|
||||
- Extra instructions (e.g. `RUN`, `COPY`, etc.) at the end of the ``final`` stage
|
||||
- Anything understood by the current ``format``
|
||||
- No
|
||||
* - ``labels``
|
||||
- Labels to tag the image
|
||||
- Pairs of key-value strings
|
||||
- No
|
||||
|
||||
.. list-table:: Configuration options specific to Singularity
|
||||
:header-rows: 1
|
||||
|
||||
* - Option Name
|
||||
- Description
|
||||
- Allowed Values
|
||||
- Required
|
||||
* - ``singularity:runscript``
|
||||
- Content of ``%runscript``
|
||||
- Any valid script
|
||||
- No
|
||||
* - ``singularity:startscript``
|
||||
- Content of ``%startscript``
|
||||
- Any valid script
|
||||
- No
|
||||
* - ``singularity:test``
|
||||
- Content of ``%test``
|
||||
- Any valid script
|
||||
- No
|
||||
* - ``singularity:help``
|
||||
- Description of the image
|
||||
- Description string
|
||||
- No
|
||||
|
||||
Once the Environment is properly configured a recipe for a container
|
||||
image can be printed to standard output by issuing the following
|
||||
command from the directory where the ``spack.yaml`` resides:
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
$ spack containerize
|
||||
|
||||
The example ``spack.yaml`` above would produce for instance the
|
||||
following ``Dockerfile``:
|
||||
|
||||
.. code-block:: docker
|
||||
|
||||
# Build stage with Spack pre-installed and ready to be used
|
||||
FROM spack/centos7:latest as builder
|
||||
|
||||
# What we want to install and how we want to install it
|
||||
# is specified in a manifest file (spack.yaml)
|
||||
RUN mkdir /opt/spack-environment \
|
||||
&& (echo "spack:" \
|
||||
&& echo " specs:" \
|
||||
&& echo " - gromacs+mpi" \
|
||||
&& echo " - mpich" \
|
||||
&& echo " concretization: together" \
|
||||
&& echo " config:" \
|
||||
&& echo " install_tree: /opt/software" \
|
||||
&& echo " view: /opt/view") > /opt/spack-environment/spack.yaml
|
||||
|
||||
# Install the software, remove unecessary deps
|
||||
RUN cd /opt/spack-environment && spack install && spack gc -y
|
||||
|
||||
# Strip all the binaries
|
||||
RUN find -L /opt/view/* -type f -exec readlink -f '{}' \; | \
|
||||
xargs file -i | \
|
||||
grep 'charset=binary' | \
|
||||
grep 'x-executable\|x-archive\|x-sharedlib' | \
|
||||
awk -F: '{print $1}' | xargs strip -s
|
||||
|
||||
# Modifications to the environment that are necessary to run
|
||||
RUN cd /opt/spack-environment && \
|
||||
spack env activate --sh -d . >> /etc/profile.d/z10_spack_environment.sh
|
||||
|
||||
|
||||
# Bare OS image to run the installed executables
|
||||
FROM centos:7
|
||||
|
||||
COPY --from=builder /opt/spack-environment /opt/spack-environment
|
||||
COPY --from=builder /opt/software /opt/software
|
||||
COPY --from=builder /opt/view /opt/view
|
||||
COPY --from=builder /etc/profile.d/z10_spack_environment.sh /etc/profile.d/z10_spack_environment.sh
|
||||
|
||||
RUN yum update -y && yum install -y epel-release && yum update -y \
|
||||
&& yum install -y libgomp \
|
||||
&& rm -rf /var/cache/yum && yum clean all
|
||||
|
||||
RUN echo 'export PS1="\[$(tput bold)\]\[$(tput setaf 1)\][gromacs]\[$(tput setaf 2)\]\u\[$(tput sgr0)\]:\w $ \[$(tput sgr0)\]"' >> ~/.bashrc
|
||||
|
||||
|
||||
LABEL "app"="gromacs"
|
||||
LABEL "mpi"="mpich"
|
||||
|
||||
ENTRYPOINT ["/bin/bash", "--rcfile", "/etc/profile", "-l"]
|
||||
|
||||
.. note::
|
||||
Spack can also produce Singularity definition files to build the image. The
|
||||
minimum version of Singularity required to build a SIF (Singularity Image Format)
|
||||
from them is ``3.5.3``.
|
||||
BIN
lib/spack/docs/dockerhub_spack.png
Normal file
BIN
lib/spack/docs/dockerhub_spack.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 88 KiB |
@@ -49,6 +49,8 @@ Spack uses a "manifest and lock" model similar to `Bundler gemfiles
|
||||
managers. The user input file is named ``spack.yaml`` and the lock
|
||||
file is named ``spack.lock``
|
||||
|
||||
.. _environments-using:
|
||||
|
||||
------------------
|
||||
Using Environments
|
||||
------------------
|
||||
|
||||
@@ -66,6 +66,7 @@ or refer to the full manual below.
|
||||
config_yaml
|
||||
build_settings
|
||||
environments
|
||||
containers
|
||||
mirrors
|
||||
module_file_support
|
||||
repositories
|
||||
|
||||
Reference in New Issue
Block a user