8 SCM/CI Workflow Integration #

With this integration, you can take advantage of source code management (SCM) systems like GitHub, GitLab or Gitea to manage your packages sources. Then, you can integrate those sources with OBS to run different workflows, for instance, to build a package and report back the result to the SCM.

In the following sections, you will find the instructions to set up the integration between SCMs and OBS.

This chapter talks in GitHub jargon to simplify the text. As constantly mentioning all the names for the same things, e.g. Pull Requests/Merge Requests, is tiresome and confusing. However, every aspect has its correspondence in GitLab and Gitea. Refer to Section 8.1.10, “Equivalence Table” for clarification of terminology.

Before you start, you need to

We support the GitHub.com and GitLab.com instances.

We also support Self-Hosted instances from GitHub, GitLab and Gitea. As long as the network connectivity works, OBS will be able to interact with that SCM.

OBS and GitHub need to talk to each other. Tokens are the way to make this happen.

8.1.4.2 How to Authenticate SCMs with OBS #

 

You have to create an OBS workflow token. Github is going to use it to trigger actions on OBS on your behalf.

8.1.4.2.1 Create Token #

 

You can create the OBS token via WebUI in Profile > Manage Your Tokens.

You can also use osc for this:

osc token --create --operation workflow --scm-token long_ascii_salad

Example of response:

<status code="ok">
    <summary>Ok</summary>
    <data name="token">long_ascii_salad</data>
    <data name="id">12345</data>
</status>

Make sure you replace long_ascii_salad with your real GitHub personal access token created in Section 8.1.4.1, “How to Authenticate OBS with SCMs”

Warning

Don't forget to keep your token secret to prevent someone else from triggering operations in your name!

8.1.4.2.2 Regenerating Secrets and Deleting Tokens #

 

If you suspect your OBS token secret was leaked, you can regenerate the secret or delete the whole token to secure it again:

a) Regenerate the token secret

Through the WebUI in Profile > Manage Your Tokens > Edit > Regenerate Token.

b) Delete the token

You can always delete your token via WebUI, in Profile > Manage Your Tokens, or with these commands:

osc token # list all your tokens

osc token --delete $token_id # remove the token with the given id

Then you can create a new one as explained in Section 8.1.4.2, “How to Authenticate SCMs with OBS” and replace it wherever you use it.

Once OBS and GitHub are allowed to speak to each other, they can start talking via webhooks.

8.1.5.2 How to Set Up a Webhook on Github #

 

Go to the project you want to set the integration on, then under Settings > Webhooks.

Figure 8.1: Webhooks on GitHub. #

 

You have to fill in the form with:

• Payload URL: https://build.opensuse.org/trigger/workflow?id=12345. Replace 12345 with the OBS token numerical ID previously obtained.

• Content type: application/json.

• Secret: uvwxyz. Replace uvwxyz with the OBS token secret string previously obtained.

• Enable SSL verification.

• Let me select individual events: Pull requests, Pushes.

8.1.5.3 How to Set Up a Webhook on GitLab #

 

Go to the project you want to set the integration on, under the Settings > Webhooks.

Figure 8.2: Webhooks on GitLab. #

 

Fill in the following fields:

• URL: https://build.opensuse.org/trigger/workflow?id=12345. Replace 12345 with the OBS token numerical ID previously obtained.

• Secret Token: uvwxyz Replace uvwxyz with the OBS token secret string previously obtained.

• Trigger: Merge request events, Push events, Tag push events.

8.1.5.4 How to Set Up a Webhook on Gitea #

 

Go to the repository you want to set the integration on, then under Settings > Webhooks.

Figure 8.3: Webhooks on Gitea. #

 

You have to fill in the form with:

• Target URL: https://build.opensuse.org/trigger/workflow?id=12345. Replace 12345 with the OBS token numerical ID previously obtained.

• HTTP Method: POST.

• POST Content Type: application/json.

• Secret: uvwxyz. Replace uvwxyz with the OBS token secret string previously obtained.

• Custom events...: Pull Request, Push.

A GitHub event occurs and OBS receives the corresponding webhook. Now is when the OBS workflows come into play.

A workflow is nothing else than a sequence of steps you want to perform in OBS. You can describe the steps to run in a YAML configuration file.

To do so, in the root directory of your GitHub repository, create a directory .obs which contains a file called workflows.yml. If you don't want to use that directory, you should check Configuration File Location out.

The content of .obs/workflows.yml could look like this:

rebuild_master:
  steps:
    - rebuild_package:
        project: home:Admin
        package: ctris
  filters:
    event: push

You can also define multiple workflows, each one needs an unique name. The following example contains two workflows: main_workflow and rebuild_master.

main_workflow:
  steps:
    - branch_package:
        source_project: OBS:Server:Unstable
        source_package: obs-server
        target_project: OBS:Server:Unstable:CI
  filters:
    event: pull_request
rebuild_master:
  steps:
    - rebuild_package:
        project: home:Admin
        package: ctris
  filters:
    event: push
    branches:
      only:
        - master

8.1.6.1 Configuration File Location #

 

By default the configuration file is fetched from the repository's target branch under .obs/workflows.yml. You can of course adjust that by editing the token configuration in OBS. The following options are available:

• Path for Workflows Configuration File allows you to adjust the path to be different than the default .obs/workflows.yml in the code repository.

• URL to Workflows Configuration File allows you to use a file that is hosted in a different place than the code repository.

Figure 8.4: Configuration File Location Options in Edit Token #

 

8.1.6.2 OBS Workflow Steps #

 

We support the following steps (the keys used in the configuration file appears surrounded with parenthesis):

• Branch a package in a project (branch_package).

• Submit a request (submit_request).

• Link a package to a project (link_package).

• Configure repositories/architectures for a project (configure_repositories)

• Rebuild a package (rebuild_package)

• Set flags for projects, packages, repositories or architectures (set_flags)

• Trigger services of a package (trigger_services)

Warning

The user the token belongs to needs to have permissions to branch a package, link packages, configure repositories/architectures, rebuild packages and trigger services of a package.

8.1.6.2.1 Branch a Package in a Project #

 

Given we have a source package called ctris coming from a source project called games, and a target project called home:jane, this step will branch that package onto the target project, keeping in mind that:

• With a pull request event, it will go to e.g.: home:jane:github:jane:ctris:PR-1/ctris. PR-1 being the pull request number.

• With a push event for commits, it will go to e.g.: home:jane/ctris-66f2acfbded89a19935ee6d481b7cf2ab95427f6. 66f2acfbded89a19935ee6d481b7cf2ab95427f6 being the SHA of the latest commit that triggered the event.

• With a push event for tags, it will go to e.g.: home:jane/ctris-release_1. release_1 being the name of the tag that triggered the event.

This is an example of a configuration file with a branch package step:

workflow:
  steps:
    - branch_package:
        source_project: games
        source_package: ctris
        target_project: home:jane

Branching a package into a project that did not exist before, for instance for a pull request event, will branch the package and set up the same repositories that the source project has. If you want to skip this and set up repositories yourself, with the configure_repositories step, set the add_repositories key to anything else than enabled.

8.1.6.2.2 Submit a Request #

 

The submit request step is the equivalent of the osc submitrequest command.

The requirements to run a submit request step are:

• There has to be a source package.

• There has to be some changes between the source package and the target package.

After the previous requirements are met, keep in mind that:

• With a pull request open event, push event, or tag_push event, it will create the submit request.

• When more commits are added to the pull request, it will supersede the request it previously created with a new request.

• With a pull_request closed event, it will revoke the request.

This is an example of a configuration file with a submit request step:

workflow:
  steps:
    - submit_request:
        source_project: games
        source_package: ctris
        target_project: home:jane_doe
        target_package: ctris                        # (optional, uses source_package if not set)
        description: 'Check out this cool package'   # (optional, uses the commit/pull request message if not set)

8.1.6.2.3 Link a Package to a Project #

 

The link package step is the equivalent of osc linkpac command.

Given a source project called devel, a source package called gcc, a target project called home:jane, and a GitHub fork called jane/gcc the step will link the package devel/gcc against:

• home:jane:github:jane:gcc:PR-1/gcc for a pull request event. PR-1 being the pull request number.

• home:jane/gcc-fae00a0ac0e5687343a60ae02bf60352002ab9aa with a push event for commits. fae00a0ac0e5687343a60ae02bf60352002ab9aa being the SHA of the latest commit that triggered the event.

• home:jane/gcc-release_1 with a push event for tags. release_1 being the name of the tag that triggered the event.

This is an example of a configuration file with a link package step:

workflow:
  steps:
    - link_package:
        source_project: devel
        source_package: gcc
        target_project: home:jane

Note

If you rely on Using Source Services to run, for instance to pick up changes from a PR with the obs_scm service, you can't make use of this step. Package links do not run the services. Use the branch_package step instead.

8.1.6.2.4 Configure Repositories/Architectures for a Project #

 

Given a project called home:jane, the step will configure a number of repositories and architectures for:

• home:jane:jane_github:repo123:PR-1 when the event is a pull request. jane_github being the username/organization which owns the SCM repository. repo123 being the name of the SCM repository. PR-1 being the pull request number.

• home:jane when the event is a commit or tag push.

Each repository needs:

• a name, e.g.: openSUSE_Tumbleweed

• a list of paths, each having a target project (e.g: openSUSE:Factory) and target repository (e.g: snapshot)

• a list of architectures to be defined for each repository. e.g.: x86_64 and i586

This is an example of a configuration file with a configure repositories step:

workflow:
  steps:
    - configure_repositories:
        project: home:jane
        repositories:
          - name: openSUSE_Tumbleweed
            paths:
              - target_project: openSUSE:Factory
                target_repository: snapshot
              - target_project: openSUSE:Tumbleweed
                target_repository: standard
            architectures:
              - x86_64
              - i586
          - name: openSUSE_Leap_15.2
            paths:
              - target_project: openSUSE:Leap:15.2
                target_repository: standard
            architectures:
              - x86_64

8.1.6.2.5 Rebuild a Package #

 

Given a project called home:Admin and a package ctris, the step will rebuild the package home:Admin/ctris.

This is an example of a configuration file with a rebuild package step.

workflow:
  steps:
    - rebuild_package:
        project: home:Admin
        package: ctris

8.1.6.2.6 Set Flags for Projects, Packages, Repositories or Architectures #

 

There are OBS-wide defaults for each flag type. This step is only necessary if you want to diverge from the defaults (see Valid flag types).

Providing the type build, the status enable and the project home:Admin, OBS will enable all builds:

• for the home:Admin:$MY_SCM_ORG:$MY_SCM_PROJECT:PR-$MY_PR_NUMBER project when the webhook event is a pull request.

• for the home:Admin project when the webhook event is a push.

Providing multiple flags is supported as noted in the configuration file below:

workflow:
  steps:
    - set_flags:
        flags:
          - type: build
            status: enable
            project: home:Admin
          - type: publish
            status: disable
            project: home:Admin

The type, status and project keys are always required. Optional keys are also available to limit the flag to a package, repository or architecture.

The project, package, repository and architecture should exist before a flag is set for them. They can be created in steps preceding a set_flags step, although this isn't necessary as long as they exist.

The type has to be one of the following values:

Valid flag types #

 

• lock (default status disable)

• build (default status enable)

• publish (default status enable)

• debuginfo (default status disable)

• useforbuild (default status enable)

• binarydownload (default status enable)

• sourceaccess (default status enable)

• access (default status enable)

The status is either disable or enable.

Take into consideration, that if the set_flags step doesn't define a flag specifically, a flag which had been set previously will preserve its value.

So with the configuration file provided below and a pull request event, builds of the home:Admin:$MY_SCM_ORG:$MY_SCM_PROJECT:PR-$MY_PR_NUMBER/ctris package will be disabled for the openSUSE_Tumbleweed repository and x86_64 architecture. For a push event, it's exactly the same, except for the package which is home:Admin/ctris-$MY_COMMIT_SHA_OR_TAG_NAME.

workflow:
  steps:
    - set_flags:
        flags:
          - type: build
            status: disable
            project: home:Admin
            package: ctris
            repository: openSUSE_Tumbleweed
            architecture: x86_64

8.1.6.2.7 Trigger Services of a Package #

 

Given a project called home:Admin and a package ctris, the step will trigger services of the package home:Admin/ctris.

Be sure to have a _service file in the package home:Admin/ctris.

This is an example of a configuration file with a trigger services step:

workflow:
  steps:
    - trigger_services:
        project: home:Admin
        package: ctris

8.1.6.3 Filters #

 

You can customize when workflows run by declaring branch or Event filters. They will make workflows run or not for specific branches/events.

You can define them in the configuration file .obs/workflows.yml. Here's an example:

workflow:
  steps:
    - branch_package:
        source_project: home:jane_doe
        source_package: ctris
        target_project: games
  filters:
    event: pull_request
    branches:
      only:
        - master
        - staging

8.1.6.3.1 Filters Delimiters: only and ignore #

 

Some steps can affect a group of elements (branches) You can use filter delimiters like only and ignore to specify which elements should be affected, or not, by the step.

The available filters delimiters are:

• only: the step only affects the elements in the list.

• ignore: the step affects all the elements except those in the list.

Note

only has precedence over ignore, so if both are defined, ignore is not considered.

This is an example to run a workflow only for the target branches master:

workflow:
  steps:
    - rebuild_package:
        project: games
        package: ctris
  filters:
    branches:
      only:
        - master

This is an example to run a workflow for all the target branches except for the branch staging:

workflow:
  steps:
    - rebuild_package:
        project: games
        package: ctris
  filters:
    branches:
      ignore:
        - staging

8.1.6.3.2 Event Filter #

 

Setting an event filter will run the workflow only for this event. The event filter doesn't accept multiple events. Documentation on the SCM events can be found here: Section 8.1.5.1, “SCM Events”.

The available event filters are:

• pull_request is for pull request events.

• merge_request is an alias for the 'pull_request' event. Introduced with workflow version 1.1 (also see Section 8.3.1, “Workflow Version Table”). .

• push is for push events related to commits.

• tag_push is for push events related to tags.

The following is an example to run a workflow only for a pull request event:

workflow:
  steps:
    - branch_package:
        source_project: games
        source_package: ctris
        target_project: home:jane_doe
  filters:
    event: pull_request

8.1.6.3.3 Branches Filter #

 

Matches target branches based on their names and runs a workflow only for those branches.

This is an example to run a workflow for all target branches, except master and final:

workflow:
  steps:
    - branch_package:
        source_project: home:jane_doe
        source_package: ctris
        target_project: games
  filters:
    branches:
      ignore:
        - master
        - final

Learn more about Filters Delimiters: only and ignore.

Note

tag_push events are not supported by the branches filter.

# The test_build workflow will branch a package based on the SCM repository name from which the webhook event came from.
test_build:
  steps:
    - branch_package:
        source_project: games
        source_package: %{SCM_REPOSITORY_NAME}
        target_project: games:CI
  filters:
    event: pull_request

Once all the steps in the workflow are done, OBS will report the build results back to GitHub.

OBS will show detailed package build status for each distribution and architecture you have set up in the configuration file.

Figure 8.5: Build Status #

 

Moreover, if your package builds several multibuild flavors, the status will have the flavor name appended to the package name:

Figure 8.6: Build Status for Several Multibuild Flavors #

 

Note

Due to a limitation, the initial "pending" build status of packages with multibuild flavors is not reported. The build status for those flavors will however still be reported when the build finishes.

For every SCM event, OBS compiles relevant information about the workflows running on the system. You can find the so-called "Workflow Runs" under the list of tokens.

Figure 8.7: Tokens list #

 

From the list of workflow runs, you can get information like:

Figure 8.8: Workflow Runs #

 

Click on each workflow run to get detailed information about it. OBS records the request received from the SCM,

Figure 8.9: Workflow Runs - Request #

 

the response sent back to the SCM,

Figure 8.10: Workflow Runs - Response #

 

and the artifacts used or generated during the run of each workflow.

Figure 8.11: Workflow Runs - Artifacts #

 

These records will help with debugging workflows. If an error occurs in any of the workflow steps, the workflow run will record error messages. And reading the artifacts will help to understand what happened behind the scenes.

Current available workflow versions and the introduced changes can be found in the versions table below.

For some of the use cases, you might want the OBS package to get the sources from the pull request in GitHub.

For this, you can make use of the existing obs-service-tar_scm service. Your package should include a properly defined _service file. obs-service-tar_scm will automatically use the sources of the pull request that triggered it.

You decide to manage your package sources from a GitHub repository. However, every time someone tries to add changes to your sources by opening a pull request, you need to verify that your package still builds for certain repositories and architectures in OBS. You can have the best of both worlds with the SCM/CI Workflow Integration Setup.

You will need:

This is obviously a good candidate to use the OBS SCM Service.

There are two different strategies to do this: branching the package or linking to it.

workflow:
  steps:
    - branch_package:
        source_project: GNOME:Factory
        source_package: gnome-shell
        target_project: home:jane:playground
  filters:
    event: pull_request

workflow:
  steps:
    - link_package:
        source_project: GNOME:Factory
        source_package: gnome-shell
        target_project: home:jane:playground
    - configure_repositories:
        project: home:jane:playground
        repositories:
          - name: openSUSE_Tumbleweed
            paths:
              - target_project: openSUSE:Factory
                target_repository: snapshot
            architectures:
              - x86_64
              - i586
          - name: openSUSE_Leap_15.2
            paths:
              - target_project: openSUSE:Leap:15.2
                target_repository: standard
            architectures:
              - x86_64
  filters:
    event: pull_request

You have a test build set up and you want it to keep up to date with the new changes you add to the PR. One way to do it, is to configure a rebuild package step with a push event filter.

You need:

The workflow configuration should be like this one:

workflow:
  steps:
    - rebuild_package:
        project: home:jane
        package: rust
  filters:
    event: push

When you branch a package, all its repositories and their architectures will be copied over. For this package, you might want to disable builds for a certain repository or architecture. This is possible with the set_flags step.

The workflow configuration should be like this one:

workflow:
  steps:
    - branch_package:
        source_project: home:jane_doe
        source_package: rust
        target_project: home:jane_doe:CI
    - set_flags:
        flags:
          - type: build
            status: disable
            project: home:jane_doe:CI
            package: rust
            architecture: x86_64

You have a software project for which you mark releases with Git tags. For every release, you want to create a package on OBS. This can be automated in a workflow with the branch_package step and the tag_push event filter. Once the workflow is in place, every tag you push to your SCM repository will branch a package on OBS and create, then build a package for the source code associated to the tag's commit. This way, your users can always install a versioned release of your software project. You can also link one of those versioned releases to another project on OBS if you need it as a dependency.

After the usual setup for OBS workflows with tokens and webhooks (see Section 8.1, “SCM/CI Workflow Integration Setup”), you will need:

The workflow configuration should be like this one:

workflow:
  steps:
    - branch_package:
        source_project: home:jane
        source_package: my_package
        target_project: home:jane:releases
  filters:
    event: tag_push

The _service file in your source package should be like:

<?xml version="1.0"?>
  <services>
    <service name="obs_scm">
      <param name="versionformat">@PARENT_TAG@</param>
      <param name="url">https://github.com/jane/my_package.git</param>
      <param name="scm">git</param>
      <param name="revision">@PARENT_TAG@</param>
      <param name="extract">my_package.spec</param>
    </service>
    <service name="set_version"/>
    <service name="tar" mode="buildtime"/>
  </services>

Here's an explanation of the services involved:

For the spec file in your SCM repository, pay attention to this:

It may happen that you have multiple repositories following the same set of workflow steps, and you would rather have one copy of the configuration file stored in a single place and applied to multiple workflows. This can be done with the combination of placeholder variables and the configuration file url setting

Let's say you have the following configuration file that works with your SCM repository called gnome-shell

workflow:
  steps:
    - branch_package:
        source_project: "test-project"
        source_package: "gnome-shell"
        target_project: "test-target-project"
  filters:
    event: pull_request

If you replace gnome-shell with %{SCM_REPOSITORY_NAME} like so:

workflow:
  steps:
    - branch_package:
        source_project: "test-project"
        source_package: "%{SCM_REPOSITORY_NAME}"
        target_project: "test-target-project"
  filters:
    event: pull_request

It will perform just as well as it did before, however now this configuration can be applied to any other OBS package in the test-project and SCM repository combination assuming they have the same name.

From here the only thing left to do would be to host this file somewhere where OBS can access it, creating a workflow token and the corresponding webhooks (following the setup instructions at Section 8.1.4, “Token Authentication”) for every SCM repository you want this configuration file to apply to, making sure you set the correct configuration url (see Section 8.1.6.1, “Configuration File Location”).

There are many other ways to use these two features in parallel, make sure to read Section 8.1.6.4, “Placeholder Variables” and Section 8.1.6.1, “Configuration File Location” to get some inspiration on how you can use them in your project.