zhangyiss/spack
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

spack containerize: allow users to customize the base image (#15028)

Browse Source 
This PR reworks a few attributes in the container subsection of
spack.yaml to permit the injection of custom base images when
generating containers with Spack. In more detail, users can still
specify the base operating system and Spack version they want to use:

  spack:
    container:
      images:
        os: ubuntu:18.04
        spack: develop

in which case the generated recipe will use one of the Spack images
built on Docker Hub for the build stage and the base OS image in the
final stage. Alternatively, they can specify explicitly the two
base images:

  spack:
    container:
      images:
        build: spack/ubuntu-bionic:latest
        final: ubuntu:18.04

and it will be up to them to ensure their consistency.

Additional changes:

* This commit adds documentation on the two approaches.
* Users can now specify OS packages to install (e.g. with apt or yum)
  prior to the build (previously this was only available for the
  finalized image).
* Handles to avoid an update of the available system packages have been
  added to the configuration to facilitate the generation of recipes
  permitting deterministic builds.
This commit is contained in:
Massimiliano Culpo
2020-11-17 20:25:13 +01:00
committed by GitHub
parent 7ffad278d3
commit 5f636fc317
13 changed files with 514 additions and 238 deletions
Download Patch File Download Diff File Expand all files Collapse all files

20
share/spack/templates/container/Dockerfile
View File

@@ -1,12 +1,19 @@
# Build stage with Spack pre-installed and ready to be used
FROM {{ build.image }}:{{ build.tag }} as builder
FROM {{ build.image }} as builder
{% if os_packages_build %}
# Install OS packages needed to build the software
RUN {% if os_package_update %}{{ os_packages_build.update }} \
&& {% endif %}{{ os_packages_build.install }} {{ os_packages_build.list | join | replace('\n', ' ') }} \
&& {{ os_packages_build.clean }}
{% endif %}
# What we want to install and how we want to install it
# is specified in a manifest file (spack.yaml)
RUN mkdir {{ paths.environment }} \
{{ manifest }} > {{ paths.environment }}/spack.yaml
# Install the software, remove unecessary deps
# Install the software, remove unnecessary deps
RUN cd {{ paths.environment }} && spack env activate . && spack install --fail-fast && spack gc -y
{% if strip %}
@@ -34,16 +41,15 @@ COPY --from=builder {{ paths.store }} {{ paths.store }}
COPY --from=builder {{ paths.view }} {{ paths.view }}
COPY --from=builder /etc/profile.d/z10_spack_environment.sh /etc/profile.d/z10_spack_environment.sh
{% if os_packages %}
RUN {{ os_packages.update }} \
&& {{ os_packages.install }}{% for pkg in os_packages.list %} {{ pkg }}{% endfor %} \
&& {{ os_packages.clean }}
{% if os_packages_final %}
RUN {% if os_package_update %}{{ os_packages_final.update }} \
&& {% endif %}{{ os_packages_final.install }} {{ os_packages_final.list | join | replace('\n', ' ') }} \
&& {{ os_packages_final.clean }}
{% endif %}
{% if extra_instructions.final %}
{{ extra_instructions.final }}
{% endif %}
{% for label, value in labels.items() %}
LABEL "{{ label }}"="{{ value }}"
{% endfor %}

34
share/spack/templates/container/singularity.def
View File

@@ -1,8 +1,17 @@
Bootstrap: docker
From: {{ build.image }}:{{ build.tag }}
From: {{ build.image }}
Stage: build
%post
{% if os_packages_build.list %}
# Update, install and cleanup of system packages needed at build-time
{% if os_package_update %}
{{ os_packages_build.update }}
{% endif %}
{{ os_packages_build.install }} {{ os_packages_build.list | join | replace('\n', ' ') }}
{{ os_packages_build.clean }}
{% endif %}
# Create the manifest file for the installation in /opt/spack-environment
mkdir {{ paths.environment }} && cd {{ paths.environment }}
cat << EOF > spack.yaml
@@ -29,7 +38,6 @@ EOF
{{ extra_instructions.build }}
{% endif %}
{% if apps %}
{% for application, help_text in apps.items() %}
@@ -52,39 +60,41 @@ Stage: final
{{ paths.environment }}/environment_modifications.sh {{ paths.environment }}/environment_modifications.sh
%post
{% if os_packages.list %}
# Update, install and cleanup of system packages
{{ os_packages.update }}
{{ os_packages.install }} {{ os_packages.list | join | replace('\n', ' ') }}
{{ os_packages.clean }}
{% if os_packages_final.list %}
# Update, install and cleanup of system packages needed at run-time
{% if os_package_update %}
{{ os_packages_final.update }}
{% endif %}
{{ os_packages_final.install }} {{ os_packages_final.list | join | replace('\n', ' ') }}
{{ os_packages_final.clean }}
{% endif %}
# Modify the environment without relying on sourcing shell specific files at startup
cat {{ paths.environment }}/environment_modifications.sh >> $SINGULARITY_ENVIRONMENT
{% if extra_instructions.final %}
{{ extra_instructions.final }}
{% endif %}
{% if runscript %}
%runscript
{{ runscript }}
{% endif %}
{% if startscript %}
%startscript
{{ startscript }}
{% endif %}
{% if test %}
%test
{{ test }}
{% endif %}
{% if help %}
%help
{{ help }}
{% endif %}
{% if labels %}
%labels
{% for label, value in labels.items() %}
{{ label }} {{ value }}