2. Developer guide

2.1. Introduction

Damona exposes a few additional commands intended for container developers. Three of them are grouped under the Developer tools heading of the standard --help output.

Build a Singularity image:

damona build --help

Check that every binary of a freshly built image actually runs:

damona check --help

Review the whole catalogue (latest version, size, base image):

damona catalog --help

Finally, damona publish uploads a finished image to Zenodo. It is hidden from --help because it requires credentials and creates permanent records:

damona publish --help

2.2. Configuration file

The Damona configuration file lives at ~/.config/damona/damona.cfg and is created automatically on first use. It follows the standard configparser INI format.

A typical file looks like:

[general]
verbose=True

[urls]
damona=https://biomics.pasteur.fr/salsa/damona/registry.txt

[zenodo]
token=APmm6p....
orcid=0000-0001
name='Cokelaer, Thomas'
affiliation='Institut Pasteur'

[sandbox.zenodo]
token=FFmbAEhQbb...
orcid=0000-0001
name='Cokelaer, Thomas'
affiliation='Institut Pasteur'

The [urls] section defines short aliases for external registry URLs. When a user runs:

damona install example --registry damona

the alias damona is replaced by its full URL. The URL must end with the file name registry.txt or registry.yaml.

The [zenodo] sections are only needed for developers who upload images; they are not required for normal use.

2.3. Publishing images on Zenodo

The goal is to assign a permanent, citable DOI to every container:

git clone git@github.com/your_fork/damona
cd damona

Consider the example tool SOFTWARE. Navigate to its recipe directory:

cd damona/software/SOFTWARE

Warning

The following commands require a registered Zenodo token and will actually create Zenodo records. Always test with the sandbox first (the default) before using --production.

Case 1 – New tool (no existing Zenodo record):

Test with the sandbox first (default), then publish for real:

damona publish SOFTWARE_1.0.0.img               # sandbox — no real DOI
damona publish SOFTWARE_1.0.0.img --production  # production — permanent DOI

Damona uploads the image, creates a new Zenodo deposit, and then prompts interactively for the binary names that the container exposes:

Binary names for 'SOFTWARE' (space or comma separated) [SOFTWARE]: software softtool

The answer is written as a binaries: field at the top of the generated registry.yaml. Review the file, then commit and push.

To skip the interactive prompt (e.g. in a script), pass --binaries directly:

damona publish SOFTWARE_1.0.0.img --production --binaries "software softtool"

Case 2 – New version of an existing tool:

damona publish SOFTWARE_2.0.0.img --production

Damona appends the new release block to the local registry.yaml and prompts for any binaries that are specific to this release (i.e. not covered by the top-level binaries: field):

Extra binaries for this release (space or comma separated, empty to skip) []:

Leave the prompt empty if no extra binaries are needed. If extra binaries are provided they are written as extra_binaries: inside the release block. Pass --extra-binaries on the command line to skip the prompt.

If the global binaries: field is currently empty in the registry, a warning is printed as a reminder to fill it in.

Case 3 – Re-publishing the same version (updated image):

If SOFTWARE_2.0.0.img was already published and you need to replace it with a corrected build, run damona publish --production again with the same filename. Damona detects the existing 2.0.0 entry in the local registry.yaml, comments it out automatically:

# 2.0.0:
#   download: https://zenodo.org/record/.../SOFTWARE_2.0.0.img
#   ...

and then appends the fresh entry below it. If the old entry had an extra_binaries: value it is offered as the default for the new entry's prompt, so no information is lost.

2.4. Repository layout

Recipes are kept in the ./damona/software/ directory, one sub-directory per tool:

damona/software/
├── registry.yaml            # global registry, auto-generated
├── fastqc/
│   ├── Singularity.fastqc_0.11.9
│   └── registry.yaml
└── salmon/
    ├── Singularity.salmon_1.3.0
    └── registry.yaml

Warning

The top-level damona/software/registry.yaml aggregates all the per-tool files and is regenerated by a pre-commit hook. Never edit it by hand; edit the per-tool registry.yaml instead.

If a tool ships only one recipe, name it:

Singularity.toolname

For multiple versions:

Singularity.toolname_x.y.z

2.5. Naming conventions

A valid Singularity recipe file must follow:

Singularity.NAME_x.y.z
Singularity.NAME_SUFFIX_x.y.z

The resulting image seen by users will appear as:

NAME:x.y.z
NAME_SUFFIX:x.y.z

Note

Names may be mixed-case in the recipe but Singularity Hub converts them to lowercase. Always use lowercase when referring to images in commands such as damona install pkgname:x.y.z.

2.6. Building an image

Test a recipe locally:

damona build Singularity.pkgname_x.y.z

This is a thin wrapper around the container command available on your system (apptainer or singularity):

apptainer build pkgname_x.y.z.img Singularity.pkgname_x.y.z

You can also build a registered recipe by name (any name shown by damona list):

damona build fastqc:0.11.9

or convert a Docker Hub image directly:

damona build docker://biocontainers/bowtie2:v2.4.1_cv1

When the source carries no version — typical of docker:// URLs — the output name cannot be guessed and must be given explicitly:

damona build docker://kapeel/hisat2 --destination hisat2_v2.0.0.img

Use --force to overwrite an image that already exists.

2.7. Checking a built image

Before publishing, verify that every binary declared in the registry is actually present and executable inside the image:

damona check ~/.config/damona/images/fastqc_0.11.9.img

Each registered binary is run inside the container and reported as found or broken. The exit code is 0 when all binaries pass and 1 if any fails, which makes the command usable in a CI job. To check a list of your own instead of the registered one:

damona check fastqc_0.11.9.img --binaries fastqc

2.8. Auditing the catalogue

damona catalog prints, for every container of the local registry, its latest version, download size, and the base image inferred from the Singularity definition file:

damona catalog

Sort the table to spot the containers worth slimming down, or to group those sharing a base image:

damona catalog --sort size
damona catalog --sort base

This is the quickest way to find containers still built on a heavy base (e.g. micromamba) that could move to a leaner one such as alpine or debian-slim.

2.9. Writing Singularity recipes

Pin the base image version to avoid silent changes:

BootStrap: docker
From: mambaorg/micromamba:2.5.0   # good

rather than:

BootStrap: docker
From: mambaorg/micromamba:latest  # avoid

Recommended %environment block to prevent locale warnings:

%environment
    LANG=C.UTF-8
    LC_ALL=C.UTF-8
    export LANG LC_ALL

Adding a %test block makes it easy to verify the image after building:

%test
    command --version

Labels and help sections are optional.

2.10. Micromamba-based recipes

A reusable micromamba base image is kept alongside the recipes (see damona/software/sequana/micromamba_2.5.0.img). Use it to install conda packages without building from scratch:

Bootstrap: localimage
From: micromamba_2.5.0.img

%post
    apt -y update && apt -y upgrade

    export PATH=/opt/conda/envs/main/bin/:$PATH
    export OPTS=" -q -c conda-forge -c bioconda -n main -y "

    micromamba install $OPTS python="3.9"
    micromamba install $OPTS "art==3.19.15"

    micromamba clean --packages -y
    micromamba clean --all -y
    rm -rf /opt/conda/pkgs/*
    rm -rf /root/.mamba /root/.conda /opt/conda/envs/main/share/doc

%environment
    export PATH=/opt/conda/envs/main/bin/:$PATH

%runscript
    art_illumina "$@"

Two details in that block matter more than they look:

  • Prepend, never append, the conda ``bin`` directory to ``PATH``. With PATH=$PATH:/opt/conda/... the Debian system interpreters (perl, python) come first and shadow the conda ones, which produces confusing "module not found" failures at run time.

  • Clean the package cache with the correct path. micromamba clean leaves tarballs behind, so the explicit rm -rf /opt/conda/pkgs/* is what actually reclaims the space — often several hundred megabytes. Note that a misspelled path here fails silently and simply ships a bloated image.

Danger

Never run rm -rf /tmp/* (or any glob under /tmp) inside %post. Apptainer bind-mounts the host /tmp into the build container, so such a command deletes the files of the user doing the build.

Adding set -e at the top of a %test block is recommended so that a failing command actually fails the test instead of being ignored.

2.11. Registry format

Each tool requires a registry.yaml file. Two equivalent layouts are supported:

Binaries per-release:

fastqc:
    releases:
      0.11.9:
        download: https://example.com/fastqc_0.11.9.img
        md5sum: abc123...
        binaries: fastqc
      0.11.8:
        download: https://example.com/fastqc_0.11.8.img
        md5sum: def456...
        binaries: fastqc

Shared binaries (applies to all releases unless overridden):

fastqc:
    binaries: fastqc
    releases:
      0.11.9:
        download: https://example.com/fastqc_0.11.9.img
        md5sum: abc123...
      0.11.8:
        download: https://example.com/fastqc_0.11.8.img
        md5sum: def456...

When a new release ships additional executables not present in older versions, declare them with extra_binaries: inside that release block. They are combined with the top-level binaries: at install time:

busco:
    binaries: busco
    releases:
      6.0.0:
        download: https://zenodo.org/record/.../busco_6.0.0.img
        md5sum: a24cabbbc9...
        filesize: 730955776
        extra_binaries: miniprot,miniprot_index
      5.4.6:
        download: https://zenodo.org/record/.../busco_5.4.6.img
        md5sum: 9707085637...
        filesize: 546848768

Marking broken releases:

If a release is found to be buggy or otherwise unsuitable for new users, mark it with broken: true. Broken releases are:

  • Hidden from damona search results

  • Skipped when auto-selecting the latest version with damona install name

  • Still installable via explicit version: damona install name:x.y.z (with a warning)

This preserves reproducibility (users can still access the version) while preventing accidental use:

fastqc:
    binaries: fastqc
    releases:
      0.11.9:
        download: https://zenodo.org/record/.../fastqc_0.11.9.img
        md5sum: abc123...
      0.11.8:
        download: https://zenodo.org/record/.../fastqc_0.11.8.img
        md5sum: def456...
        broken: true  # Hide from search, skip auto-pick

damona publish prompts for extra_binaries interactively and writes the field automatically when a non-empty value is given.

The download value can be:

  1. A direct HTTPS URL to a .img or .sif file.

  2. A Docker Hub reference, e.g. docker://biocontainers/hisat2:v2.1.0-2-deb_cv1.

Example with a Docker source:

hisat2:
    releases:
      2.1.0:
        download: docker://biocontainers/hisat2:v2.1.0-2-deb_cv1
        binaries: hisat2 hisat2-build
        md5sum: e680e5ab181e73a8b367693a7bd71098

2.12. Where are images stored?

Since December 2021, Damona stores all images with a DOI on Zenodo. Previously some images were hosted on Sylabs Cloud Library.

You can always use the bundled offline registry (no internet required) by passing --local-registry-only:

damona search fastqc --local-registry-only
damona install fastqc --local-registry-only

2.13. Automatic README generation

The script build_readme.py in damona/software/ parses a registry.yaml and produces a standardised README.md. Run it from the tool's recipe directory to create or update the README:

cd damona/software/fastqc
python ../build_readme.py

2.14. Setting up the pre-commit hook

The repository uses pre-commit to keep the global registry up-to-date automatically on every commit:

pip install pre-commit
pre-commit install
git commit .