.. _templates:

CI Templates
============

The CI templates are a set of Gitlab CI job *templates* that can be
integrated into your ``.gitlab-ci.yml`` file. These templates simplify
building and re-using container images for the project's
registry.

To make use of the templates, you need to

- :ref:`include the template <templates_including>` and
- :ref:`define a job extending the template <templates_extends>`

The CI templates provide a set of templates per distribution and you need to
:ref:`include <templates_including>` the one for the distribution(s) you
want to use. The examples below use the Fedora templates.

.. _templates_why:

Why use the CI templates?
-------------------------

With the CI templates you can easily build a **persistent container image**.
Images are identified by a **unique tag** that allows for them to be used
in the actual CI jobs.

So you can say *"I want a Fedora 31 container with packages X, Y, Z
installed"*, tag this with *"2020-02-03.0"* for the current date (best practice
but not required) and that image can be used from all tests. To build a
new container or rebuild the existing container with updated packages,
simply change the tag in your ``.gitlab-ci.yml``.

Merge request re-use the same container image for CI. The CI templates will
take care of the rest. This gives you **reproducible test results** as all
merge requests will use the same container images.

Where a merge request changes the tag, a new container is
built **for that merge request** and this merge request runs CI with the new
container image. All other merge requests are unaffected. Only once the
merge request has been merged into the upstream project will future merge
requests use the new container image.

All this is available with minimal boilerplate - you specify the
distribution, version and the packages to install and the templates do the
rest.

.. _templates_including:

Including the CI templates
--------------------------
There are two ways of including a template in your project, depending on
whether your project is :ref:`hosted on gitlab.freedesktop.org <templates_including_fdo_projects>`
or :ref:`hosted elsewhere <templates_including_outside_projects>`. In both
cases, you should use a specific git sha of the files you want to include
- the CI templates do not use version numbers.

.. warning:: You can use ``master`` as git ref but we do not recommend this.
             Using a specific git sha protects **you** from unplanned CI failures caused by
             changes in the CI templates.

.. _templates_including_fdo_projects:

Projects hosted on gitlab.freedesktop.org
....................................................

If your project
is hosted on ``gitlab.freedesktop.org`` you can include it as follows:

.. code-block:: yaml

   .templates_sha: &templates_sha 123456deadbeef

   include:
     - project: 'freedesktop/ci-templates'     # the project to include from
       ref: *templates_sha                 # git ref of that project
       file: '/templates/fedora.yml'       # the actual file to include
     - project: 'freedesktop/ci-templates'
       ref: *templates_sha
       file: '/templates/alpine.yml'

   # rest of your .gitlab-ci.yml goes here

The above snippet first defines the git sha of the CI templates we want to
include as a YAML anchor. It then includes the Fedora and Alpine templates
from the CI templates repository of that specific sha. Using a YAML anchor
is recommended to avoid duplication.

You can specify different shas for different files though it is not
something we recommend. You can specify any valid git ref (e.g.
``master``) though we recommend that you use a specific sha.

For more information on the ``include:`` statement, see the `GitLab documentation
<https://docs.gitlab.com/ce/ci/yaml/#include>`__

.. _templates_including_outside_projects:

Projects not hosted on gitlab.freedesktop.org
.............................................

If youre project is not hosted on ``gitlab.freedesktop.org`` you can
use the CI templates as follows.

.. code-block:: yaml

   include:
     - remote: 'https://gitlab.freedesktop.org/freedesktop/ci-templates/-/raw/123456deadbeef/templates/fedora.yml'
     - remote: 'https://gitlab.freedesktop.org/freedesktop/ci-templates/-/raw/123456deadbeef/templates/alpine.yml'

   # rest of your .gitlab-ci.yml goes here

The above snippets links to the files directly using the git sha.

You can specify different shas for different files though it is not
something we recommend. You can specify any valid git ref (e.g.
``master``) though we recommend that you use a specific sha.


For more information on the ``include:`` statement, see the `GitLab documentation
<https://docs.gitlab.com/ce/ci/yaml/#include>`__

.. _templates_extends:

Extending the template jobs
---------------------------

All jobs provided by the CI templates use a naming scheme in the form
``.fdo.<type>@<distribution>``. They start with a dot (``.``) and thus
do not get invoked by the GitLab CI runners. To make use of them, use
``extends:``.

.. code-block:: yaml

    # include statements go here

    myjob:
      extends: .fdo.container-build@fedora@x86_64
      stage: somestage
      variables:
        FDO_DISTRIBUTION_VERSION: 31
        FDO_DISTRIBUTION_PACKAGES: 'curl wget valgrind'


In the (incomplete) example above a job called ``myjob`` is defined to be
invoked in the ``somestage`` stage (user's choice). It extends the
``.fdo.container-build@fedora@x86_64`` template (see
:ref:`templates_building_containers`). The ``variables`` will be
used by the CI template to generate the correct container image.

For more information on the ``extends:`` statement, see the `GitLab documentation
<https://docs.gitlab.com/ce/ci/yaml/#extends>`__


.. _templates_building_containers:

Building container images
-------------------------

The CI templates provide two ways of building a container image:
``container-build`` for a normal container and ``qemu-build``
for building an image that can be run through QEMU on a KVM-enabled host.
For projects without specific hardware interactions the
``container-build`` is sufficient.

As the name implies, this builds a container image if it does not already
exist. At runtime, it will check your image repository (i.e. the GitLab
``username/project``) for the container image and then the upstream
repository.

If the image does not exist in your repository but it does exist
upstream, the image is copied to your repository. Thus, even if upstream
changes the image later, you have a copy of that image.

If the image does not exist upstream, it is built in **your** repository.
Once your merge request is merged, **the image will be rebuilt for the
upstream repository**.

.. note:: If you want to **force** a container build, set the
          variable ``FDO_FORCE_REBUILD``.

Below is an example on how to build a container. To avoid repetition of
boilerplate code, we define a template for the distribution we want to build
on and re-use that template to fill in the variables for us where required.

.. code-block:: yaml

    include:
    - project: 'freedesktop/ci-templates'
      ref: 12345deadbeef
      file: '/templates/fedora.yml'


    # Let's define some stages to get the correct order of jobs.
    stages:
      - prep
      - test

    variables:
      # The upstream repository path on gitlab.freedesktop.org to check
      # for existing container images.
      FDO_UPSTREAM_REPO: some/path

    # A simple template so we only have one place where we need to
    # define the Fedora version and the image tag
    # Using a date as tag is best practice but it can be any string
    # allowed by the GitLab registry.
    .myproject.fedora:30:
      variables:
        FDO_DISTRIBUTION_VERSION: 31
        FDO_DISTRIBUTION_TAG: '2020-03-10.0'


    # A job to build a Fedora 31 container with valgrind and gcc
    # installed.
    #
    # You must not define script: in this job, it is used by the
    # container-build template.
    build-fedora-container:
      extends:
      - .fdo.container-build@fedora@x86_64     # the CI template
      - .myproject.fedora:30                   # our template job above
      stage: prep
      variables:
        # Packages to install on the container
        FDO_DISTRIBUTION_PACKAGES: "valgrind gcc"


    # The test job for your project. Extending from the CI Templates
    # .fdo.distribution-image makes it use the same image we built above.
    mytest:
      extends:
      - .fdo.distribution-image@fedora
      - .myproject.fedora:30
      stage: test
      script:
        # FDO_DISTRIBUTION_NAME is set by the distribution-image job.
        # It should be considered read-only
        - echo "Hello world in $FDO_DISTRIBUTION_NAME"

.. warning:: The ``.fdo.qemu-build`` template uses an arch-specific
             suffix. See :ref:`templates_multiarch` for important details.

.. _templates_building_qemu_images:

Building QEMU-capable container images
......................................

Some tests need to run in a virtual machine instead of a container. For
those, qemu-capable images can be prepared with the CI templates. The
templates are identical to the ones shown above but use the term ``qemu``
instead of ``container``. The above example thus becomes:

.. code-block:: yaml

    # A job to build a Fedora 31 qemu image with valgrind and gcc
    # installed.
    #
    # You must not define script: in this job, it is used by the
    # qemu-build template.
    build-fedora-vm:
      extends:
      - .fdo.qemu-build@fedora@x86_64   # the CI template
      - .myproject.fedora:30            # our template job above
      stage: prep
      variables:
        # Packages to install on the vm
        FDO_DISTRIBUTION_PACKAGES: "valgrind gcc"

.. warning:: The ``.fdo.qemu-build`` template uses an arch-specific
             suffix. See :ref:`templates_multiarch` for important details.

Once built, the container image provides the script ``/app/vmctl start``
to start the virtual machine. The VM is configured to accept ssh connection
and aliased as host ``vm``. Commands can be run on the virtual machine with
the ``/app/vmctl exec`` helper.

.. code-block:: yaml

    # The test job for your project. Extending from the CI Templates
    # .fdo.distribution-image makes it use the same image we built above.
    mytest:
      extends:
      - .fdo.distribution-image@fedora
      - .myproject.fedora:30
      stage: test
      script:
        # start the VM. This also sets up ssh/scp to connect to "vm"
        # correctly.
        - /app/vmctl start
        # copy our workspace to the VM
        # The quotes are required to stop the ':' from parsing as yaml
        - scp -r $PWD "vm:"
        # We don't want any failed commands to exit our script until VM
        # cleanup has been completed.
        - set +e
        # run test-command on the VM and create the .success file if it
        # succeeds
        - /app/vmctl exec "cd $CI_PROJECT_NAME ; test-command" && touch .success
        # copy any test results from the VM to our container so we can
        # save them as artifacts
        - scp -r vm:$CI_PROJECT_NAME/test-results.xml .
        # shut down the VM
        - /app/vmctl stop
        # VM cleanup is complete, any command failures now should result in
        # a CI failed job
        - set -e
        # our CI script exit code should match the test command exit status
        - test -e .success || exit 1

Noteworthy in the above is that the actual test command and subsequent VM
cleanup are wrapped by a ``set +e`` and ``set -e`` call. This ensures that
any failed call does not immediately terminate the CI job but allows for the
proper cleanup of the VM. The ``.success`` file is what determines the
actual success status of the CI job.

.. templates_container_labels:

Image labels
------------

Images built with the CI templates have a number of labels set that can be
used by jobs or the GitLab setup itself. The example below shows how to
access the labels from a CI job using `skopeo
<https://github.com/containers/skopeo>`__ and `jq
<https://stedolan.github.io/jq/>`__:

.. code-block:: yaml

    check label:
      extends:
        - .myproject.fedora:30            # our template job above
        - .fdo.distribution_image@fedora
      # We don't want/need to use the actual image we built earlier, we can use
      # any image that has skopeo and jq, or install those as part of
      # script:
      image: any-image-with-skopeo-and-jq
      script:
        # FDO_DISTRIBUTION_IMAGE still has indirections
        - DISTRO_IMAGE=$(eval echo ${FDO_DISTRIBUTION_IMAGE})
        # retrieve the infos from the registry (once)
        - JSON_IMAGE=$(skopeo inspect docker://$DISTRO_IMAGE)

        # Parse the the pipeline_id label
        - IMAGE_PIPELINE_ID=$(echo $JSON_IMAGE | jq -r '.Labels["fdo.pipeline_id"]')

        # If the image was built as part of this pipeline, the image's pipeline
        # ID is the same as the current pipeline ID.
        # This can be used to poke other projects to rebuild dependent images.
        - if [[ x"$IMAGE_PIPELINE_ID" == x"$CI_PIPELINE_ID" ]]; then
              echo "Image was built in this pipeline"
          fi

.. note:: Extending from ``.fdo.distribution_image@fedora`` provides
          ``FDO_DISTRIBUTION_IMAGE``. We do not actually use the image
          itself, any image with ``skopeo`` and ``jq`` will work here.

The labels currently set by the CI templates are as follows:

.. list-table::

 * - Image Label
   - Value
 * - ``fdo.pipeline_id``
   - ``$CI_PIPELINE_ID``
 * - ``fdo.job_id``
   - ``$CI_JOB_ID``
 * - ``fdo.commit``
   - ``$CI_COMMIT_SHA``
 * - ``fdo.project``
   - ``$CI_PROJECT_PATH``
 * - ``fdo.upstream-repo``
   - ``$FDO_UPSTREAM_REPO`` (optional)
 * - ``fdo.expires-after``
   - ``$FDO_EXPIRES_AFTER`` (optional)

For the values starting with ``CI_``, see the `GitLab
environment variables documentation
<https://docs.gitlab.com/ee/ci/variables/predefined_variables.html>`__

As in the example above, the image's ``fdo.pipeline_id`` can be used to
check if an image was built as part of the current pipeline.

.. _templates_deleting_containers:

Deleting container images
-------------------------

Unfortunately, deleting container images is nontrivial with templates and it
requires extra authentication tokens. Use the
:ref:`ci-fairy tool <ci-fairy>` for this task:

.. code-block:: yaml

    delete-containers:
      extends:
       - .fdo.distribution-image@fedora
       - .myproject.fedora:30
      stage: cleanup
      image: golang:alpine
      before_script:
        - apk add python3 git
        - pip3 install git+http://gitlab.freedesktop.org/freedesktop/ci-templates
      script:
        # Go to your Profile, Settings, Access Tokens
        # Create a personal token with 'api' scope, copy the value.
        # Go to CI/CD, Schedules, schedule a new monthly job (or edit the existing one)
        # Define a variable of type File named AUTHFILE. Content is that token
        # value.
        #
        # This example assumes that you want to delete all but the current tag
        - ci-fairy -v --authfile $AUTHFILE delete-image
                --repository $FDO_DISTRIBUTION_NAME/$FDO_DISTRIBUTION_VERSION
                --exclude-tag $FDO_DISTRIBUTION_TAG
      only:
        - schedules

This is a job to run container cleanup on a `schedule job
<https://docs.gitlab.com/ce/ci/pipelines/schedules.html>`__. We get
``$FDO_DISTRIBUTION_NAME`` by extending ``.fdo.distribution-image@fedora``
but since we only need to run a simple python tool, we can just run off a
``golang:alpine`` image.
All other variables are courtesy of ``.myproject.fedora:30`` (see
:ref:`templates_extends`)

Because of restrictions in GitLab, this can only be run with an API token,
``CI_JOB_TOKEN`` does not have permissions to delete images.

The ``ci-fairy`` command as run here will delete all images in the
``fedora/30`` image repository, excluding the one with the tag ``2020-03-10.0``.

.. _templates_multiarch:

Handling multi-arch images
--------------------------

The ``.fdo.container-build`` and ``.fdo.qemu-build`` templates use an
arch-specific suffix (``@x86_64`` or ``@aarch64``) to build for the
appropriate architecture. This arch-specific suffix is not encoded in the
image name, and a potential pitfall when building identical images for
multiple architectures is that those images overwrite each other. This
example illustrates the problem:

.. code-block:: yaml

    # DO NOT USE THIS SNIPPET. This example illustrates a bug

    .fedora:
      variables:
        FDO_DISTRIBUTION_VERSION: 32
        FDO_DISTRIBUTION_TAG: '2020-11-12.0'

    # uses distribution name, version and tag to store the image in the
    # registry
    build-x86:
      stage: prep
      extends:
        - .fdo.container-build@fedora@x86_64
        - .fedora

    # uses the same distribution name, version and tag to store the image
    # in the registry, potentially overwriting the other image.
    # if this job runs after the build-x86 job completed, it does nothing
    # because an image with the given tag is already in the registry
    build-arm:
      stage: prep
      extends:
        - .fdo.container-build@fedora@aarch64
        - .fedora

    # this job runs on whichever image got stored in the registry
    run-image:
      stage: build:
      extends:
        - .fdo.distribution-image@fedora
        - .fedora
      script:
        - echo "I don't know which image I'm running on"

Both ``build-`` jobs produce the same image tag and which image ends up in
the registry depends on the (non-deterministic) order the jobs are run
and/or completed.

Where multi-arch jobs are required, the arch must be encoded in the image
name using ``FDO_DISTRIBUTION_TAG`` (or ``FDO_REPO_SUFFIX``):

.. code-block:: yaml

    .fedora:
      variables:
        FDO_DISTRIBUTION_VERSION: 32
        BASE_TAG: "2020-11-12.0"

    .fedora-x86:
      extends:
        - .fedora
      variables:
        FDO_DISTRIBUTION_TAG: "x86_64-$BASE_TAG"

    .fedora-arm:
      extends:
        - .fedora
      variables:
        FDO_DISTRIBUTION_TAG: "aarch64-$BASE_TAG"

    build-x86:
      stage: prep
      extends:
        - .fdo.container-build@fedora@x86_64
        - .fedora-x86
      variables:
        FDO_DISTRIBUTION_TAG: "x86_64-$BASE_TAG"

    build-arm:
      stage: prep
      extends:
        - .fdo.container-build@fedora@aarch64
        - .fedora-arm
      variables:
        FDO_DISTRIBUTION_TAG: "aarch64-$BASE_TAG"

    .run-image:
      stage: build:
      extends:
        - .fdo.distribution-image@fedora
      script:
        - echo "I'm running on $FDO_DISTRIBUTION_TAG"

    run-x86:
      extends:
        - .run-image
        - .fedora-x86
      variables:
        FDO_DISTRIBUTION_TAG: "x86_64-$BASE_TAG"

    run-arm:
      extends:
        - .run-image
        - .fedora-arm
      variables:
        FDO_DISTRIBUTION_TAG: "aarch64-$BASE_TAG"

:ref:`Templating the .gitlab-ci.yml <ci-fairy-templating>` can ease the
maintenance of such pipelines.