How do I create my own Singularity containers?

https://slides.com/mbhall88/making-containers

Michael Hall - Iqbal Group (EBI)


GitHub followers Twitter Follow

Making containers

Writing a recipe

https://www.sylabs.io/guides/3.0/user-guide/definition_files.html

Writing a recipe

Sometimes called a definition file.

Tells Singularity how to build the container.

 

  • Base OS
  • Software to install
  • Environment variables
  • Files to add
  • Metadata

Writing a recipe

Bootstrap: debootstrap
OSVersion: bionic
MirrorURL: http://us.archive.ubuntu.com/ubuntu/

%help
  A container with a bunch of fun programs installed.

%environment
  export LC_ALL=C.UTF-8
  export LANG=C.UTF-8
  export PATH=/usr/games:"$PATH"

%post
  apt update
  apt install -y software-properties-common
  apt-add-repository universe
  apt update
  DEBIAN_FRONTEND=noninteractive apt install -y cmatrix cowsay fortune 
  echo "export PATH=/usr/games:$PATH" >> "$SINGULARITY_ENVIRONMENT"

%test
  export PATH=/usr/games:"$PATH"
  command -v cowsay >/dev/null 2>&1 || { echo >&2 "I require cowsay but it's not installed.  Aborting."; exit 1; }
  command -v fortune >/dev/null 2>&1 || { echo >&2 "I require fortune but it's not installed.  Aborting."; exit 1; }
  command -v cmatrix >/dev/null 2>&1 || { echo >&2 "I require cmatrix but it's not installed.  Aborting."; exit 1; }

%runscript
  if [ $# -eq 0 ]
    then
      fortune | cowsay
  else
    cowsay "$@"
  fi

%labels
  Author Michael Hall
  Version v0.0.1

https://git.embl.de/moscardo/singularity-training-2019/blob/master/recipes/Singularity.fun

Writing a recipe

Bootstrap: debootstrap
OSVersion: bionic
MirrorURL: http://us.archive.ubuntu.com/ubuntu/

Header

Tells Singularity the base operating system you would like your container to be built with.

Writing a recipe

Header

Other options include:

Bootstrap: docker
From: tensorflow/tensorflow

https://www.sylabs.io/guides/3.0/user-guide/definition_files.html#header

Bootstrap: shub
From: mbhall88/Singularity_recipes:template

Writing a recipe

Sections

Executed during the build process. If any step in any section fails, so does the build.

https://www.sylabs.io/guides/3.0/user-guide/definition_files.html#sections

%help

Useful for describing to others what your container is intended to do

https://www.sylabs.io/guides/3.0/user-guide/definition_files.html#help

%help
  A container with a bunch of fun programs installed.
singularity run-help fun.sif

%environment

Define environment variables that will be available at runtime (i.e not available during build)

Can also add to environment with $SINGULARITY_ENVIRONMENT

https://www.sylabs.io/guides/3.0/user-guide/environment_and_metadata.html#environment-and-metadata 

%environment
  export LC_ALL=C.UTF-8
  export LANG=C.UTF-8
  export PATH=/usr/games:"$PATH"
echo "export foo=bar" >> "$SINGULARITY_ENVIRONMENT"

%post

This is what runs after the base OS is installed. Use this to install software, download data, create directories etc.

https://www.sylabs.io/guides/3.0/user-guide/definition_files.html#post 

%post
  apt update
  apt install -y software-properties-common
  apt-add-repository universe
  apt update

  DEBIAN_FRONTEND=noninteractive apt install -y cmatrix cowsay fortune
  echo "export PATH=/usr/games:$PATH" >> "$SINGULARITY_ENVIRONMENT"

%test

Runs at the end of the build process. I STRONGLY encourage you to put tests in your definition.

 

Use --notest to skip tests 😒

https://www.sylabs.io/guides/3.0/user-guide/definition_files.html#test 

%test
  export PATH=/usr/games:"$PATH"
  command -v cowsay >/dev/null 2>&1 || \
    { echo >&2 "I require cowsay but it's not installed.  Aborting."; exit 1; }
  command -v fortune >/dev/null 2>&1 || \
    { echo >&2 "I require fortune but it's not installed.  Aborting."; exit 1; }
  command -v cmatrix >/dev/null 2>&1 || \
    { echo >&2 "I require cmatrix but it's not installed.  Aborting."; exit 1; }
singularity test fun.simg

# build without tests using --notest
sudo singularity build --notest fun.sif Singularity.fun

%runscript

Is run when invoked with the singularity run command or executed with ./my.sif

https://www.sylabs.io/guides/3.0/user-guide/definition_files.html#runscript 

%runscript
  if [ $# -eq 0 ]
    then
      fortune | cowsay
  else
    cowsay "$@"
  fi

%runscript

https://www.sylabs.io/guides/3.0/user-guide/definition_files.html#runscript 

./fun.sif Holy cow!

 ___________
< Holy cow! >
 -----------
        \   ^__^
         \  (oo)\_______
            (__)\       )\/\
                ||----w |
                ||     ||
singularity run fun.sif

 ___________________________
< Condense soup, not books! >
 ---------------------------
        \   ^__^
         \  (oo)\_______
            (__)\       )\/\
                ||----w |
                ||     ||

%labels

Used to add metadata to the container with a name-value pair format.

 

Inspect the metadata of a container

https://www.sylabs.io/guides/3.0/user-guide/definition_files.html#labels

%labels
  Author Michael Hall
  Version v0.0.1
singularity inspect fun.sif

{
	"Author": "Michael Hall",
	"Version": "v0.0.1",
	"org.label-schema.build-date": "Tuesday_19_February_2019_11:38:35_UTC",
	"org.label-schema.build-size": "611MB",
	"org.label-schema.schema-version": "1.0",
	"org.label-schema.usage": "/.singularity.d/runscript.help",
	"org.label-schema.usage.singularity.deffile": "Singularity.template",
	"org.label-schema.usage.singularity.deffile.bootstrap": "shub",
	"org.label-schema.usage.singularity.deffile.from": "mbhall88/Singularity_recipes:template",
	"org.label-schema.usage.singularity.deffile.mirrorurl": "http://us.archive.ubuntu.com/ubuntu/",
	"org.label-schema.usage.singularity.deffile.osversion": "bionic",
	"org.label-schema.usage.singularity.runscript.help": "/.singularity.d/runscript.help",
	"org.label-schema.usage.singularity.version": "3.0.3-1.el7"
}

Further reading...

Apps

https://www.sylabs.io/guides/3.0/user-guide/definition_files.html#apps

Best practices

https://www.sylabs.io/guides/3.0/user-guide/definition_files.html#best-practices-for-build-recipes

Recipe development cycle

Building a container

sudo singularity build <image name> <target>

Need root (sudo) access to build containers

https://www.sylabs.io/guides/3.0/user-guide/build_a_container.html

sudo singularity build <image name> <target>

<target> can be:

  • URI beginning with library:// to build from the Container Library
  • URI beginning with docker:// to build from Docker Hub
  • URI beginning with shub:// to build from Singularity Hub
  • path to a existing container on your local machine
  • path to a directory to build from a sandbox
  • path to a Singularity definition file 
sudo singularity build <image name> <target>

<image name> should have the .sif ( Singularity Image File) extension. 

sudo singularity build fun.sif Singularity.fun

Versions prior to 3.0 use .simg

Playing in a sandbox with a shell

Container development cycle

https://www.sylabs.io/guides/3.0/user-guide/build_a_container.html#creating-writable-sandbox-directories

Create a clean container and open a blank text file

Bootstrap: debootstrap
OSVersion: bionic
MirrorURL: http://us.archive.ubuntu.com/ubuntu/

%post
    apt update
    apt install -y software-properties-common
    apt-add-repository universe
    apt update

Singularity.template

Container development cycle

https://www.sylabs.io/guides/3.0/user-guide/build_a_container.html#creating-writable-sandbox-directories

Create a clean container and open a blank text file

sudo singularity build --sandbox container-dev Singularity.template

Container development cycle

https://www.sylabs.io/guides/3.0/user-guide/build_a_container.html#creating-writable-sandbox-directories

Shell into the sandbox

sudo singularity shell --writable container-dev

# should see the prompt change to something like
Singularity container-dev:~>

If you don't use --writable you won't be able to install anything or do anything that basically changes the size of the container

Container development cycle

https://www.sylabs.io/guides/3.0/user-guide/build_a_container.html#creating-writable-sandbox-directories

Shell into the sandbox

Warning: The directory /root from you local machine will be mounted in the sandbox. So anything you do in the sandbox in that directory will also be reflected in the /root directory locally. 

Fix: Move out of /root within the sandbox and do all of your work there. I personally tend to use /usr/local or you could create a new directory altogether (but outside /root) e.g /sandbox

Install conda and bwa

wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -O miniconda3.sh
apt install -y wget
bash miniconda3.sh -b -p "$HOME"/miniconda
conda list
export PATH="$HOME/miniconda/bin:$PATH"

https://conda.io/projects/conda/en/latest/user-guide/install/macos.html#install-macos-silent

source $HOME/miniconda/bin/activate

wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -O miniconda3.sh

Install conda and bwa

conda install -y -c bioconda bwa

https://conda.io/projects/conda/en/latest/user-guide/install/macos.html#install-macos-silent

Exercise

Turn it into a recipe, build, test.

 

https://git.embl.de/moscardo/singularity-training-2019/blob/master/recipes/Singularity.bwa

Exercise

Choose a software program and make a singularity container for it.

Made with Slides.com