3 OBS Concepts

We describe here the high level concepts how Open Build Service is designed, manages it's content and is supposed to work.

3.1 Project organization

All sources and binaries which are hosted inside of OBS are organized in projects. A project is the container defining a larger task. It defines who is working there

3.1.1 Project meta data

A project gets configured in the project /source/$PROJECT/_meta path. It can be edited in web interface in the RAW Config tab or via command line via

osc meta prj -e $PROJECT

This file contains

  • generic description data in title and description elements.
  • An ACL list of users and groups connected with a role. The maintainer role defines the list of users permitted to commit changes to the project.
  • A number of flags, controlling the build and publishing process and possible read access protections.
  • A list of repositories to be created. This list defines what other repositories should be used, which architectures shall be built and build job scheduling parameters.

The following flags can be used to control the behavior of a package or project. Most of them can also be limited to specified repositories or architectures.

  • build flag defines if package sources should get build. If enabled, it signals the scheduler to trigger server side builds based on events like source changes, changes of packages used in the build environment or manual rebuild triggers. A local build via CLI is possible independent of this flag. Default is enabled.
  • publish can be used to enable or disable publishing the build result as repository. This happens after an entire repository has finished build for an architecture. A publish also gets triggered, when the publish flag is enabled after a repository finished the build. Default is enabled.
  • debuginfo can be used to modify the build process to create debuginfo data along with the package build for later debugging purposes. A flag change is not triggering rebuilds, it just affects the next build. Default is disabled.
  • useforbuild is used to control if a built result shall be copied to the build pool. This means it will get used for other builds in their buildenvironment. When this is disabled, the build has no influence on builds of other packages using this repository. In case a former build exists the old binaries will be used. Disabling this flag also means that "wipe" commands to remove binary files will have no effect on the build pool. A flag change is not triggering rebuilds, it just affects the next build. Default is enabled.
  • access flag can be used to hide an entire project. This includes binaries and sources. It can only be used at project creation time and can just be enabled (making it public again) afterwards. This flag can only be used on projects. Default is enabled.
  • sourceaccess flag can be used to hide the sources, but still show the existence of a project or package. This also includes debug packages in case the distribution is supporting this correctly. This flag can only be used at package creation time. There is no code yet which is checking for possible references to this package. Default is enabled.
  • downloadbinary permission still exists like before. However, unlike "access" and "sourceaccess" this is not a security feature. It is just a convenience feature, which makes it impossible to get the binaries via the API directly. But it is still possible to get the binaries via build time in any case. Default is enabled.

3.1.2 Project build configuration.

A project gets configured in the project /source/$PROJECT/_config path. It can be edited in web interface in the Project Config tab or via one of the following command lines

osc meta prjconf -e $PROJECT
osc co $PROJECT _project

This file contains information on how to setup a build environment.

3.1.3 Project build macro configuration

The macro configuration is part of the build configuration in /source/$PROJECT/_config. It can be added at the end behind a Macro:

3.1.4 An OBS Package

An OBS Package is a sub name space below a project. It contains the specification of a single package build for all specified repositories.

3.2 The API

...

3.3 The OBS Interconnect

The OBS interconnect is a mechanism to connect two OBS instances. All content, including sources and binary build results will be available in the connecting instance. Unlike other methods the instances will also notify each other about changes.

3.4 Download on Demand Repositories (DoD)

3.4.1 Motivation

In a DoD repository external software repositories can be configured which are used for dependency resolution and where packages will be downloaded at build time. A DoD repository has some main advantages in comparison to binary import projects:

  • less disk usage as only really required packages will be downloaded
  • automatic package updates when new upstream releases are availible
  • simple to configure in project meta without need of shell access to repo servers

In download repotype's where package checksums can be verified (e.g. susetags, rpmmd and deb), it`s recommended to use a mirror server url in <download> in order to reduce traffic on the master server and configure a <master> with an https url and a sslfinger in order to avoid man in the middle attacks by peer verification.

3.4.2 XML Document Hirarchy

<project>
  <repository>
    <download>
      <master/> (optional)
      <pubkey/> (optional)
    </download>
  </repository>
</project>

3.4.3 The Daemon

The bs_dodup daemon periodically checks for new meta data in remote repositories. This daemon can be enabled for startup with the command

            systemctl enable obsdodup.service

and can be started with

            systemctl start obsdodup.service

3.4.4 The <download> element

mandatory attributes:

  • arch
  • url
  • repotype

3.4.5 The <master> subelement

The <master> tag as shown in the rpmmd example below is optional but strongly recommended for security reasons.

Verification is supported in the following repotype`s

  • susetags
  • rpmmd
  • deb

This option could be defined by any valid url ( http and https ) to the origin of the repository but it´s stronly ecommended to use https with a sslfingerprint to bs_dodup possibility to verify it`s peeer in order to avoid Man in the middle attacks. The download url can be a mirror as we validate package checksums found in repo data.

You can easily query the ssl fingerprint of a remote server by calling the following command:

openssl s_client -connect <host>:<port> < /dev/null 2>/dev/null | openssl x509 -fingerprint -noout

3.4.6 The <pubkey> subelement

The pubkey element contains one or more gpg public keys in order to verify repository information but not packages. For an example have a look at the repotype "deb" documentation below.

3.4.7 Repository Types

3.4.7.1 YAST sources (susetags)

Example:

<project name="My::SuSE::CD">

  ...

  <repository name="standard">
    <download arch="x86_64" url="http://mirror.example.org/path/to/iso" repotype="susetags" />
    <download arch="i586" url="http://mirror.example.org/path/to/iso" repotype="susetags" />
    <arch>x86_64</arch>
    <arch>i586</arch>
  </repository>
</project>

3.4.7.2 RPM sources (rpmmd)

Example:

<project name="Fedora:Rawhide">

  ...

  <repository name="standard">
    <download arch="x86_64" url="http://mirror.example.org/fedora/rawhide/x86_64/os" repotype="rpmmd">
      <master url="https://master.example.org/whereever/fedora/rawhide/x86_64/os" sslfingerprint="sha256:0a64..0303"/>
    </download>
    <download arch="i586" url="http://mirror.example.org/fedora/rawhide/i386/os" repotype="rpmmd">
      <master url="https://master.example.org/whereever/fedora/rawhide/i386/os" sslfingerprint="sha256:0a64..0303"/>
    </download>
    <arch>x86_64</arch>
    <arch>i586</arch>
  </repository>
</project>

3.4.7.3 Apt Repository (deb)

Apt supports two repository types, flat repositories and distribution repositories.

The download url syntax for them is:

  • <baseurl>/<distribution>/<components>
  • <flat_url>/.[/<components>]

You can specify multiple components seperated by a comma.

An empty components string is parsed as "main".

Example:

<project name="Debian:8">

  ...

  <repository name="ga">
    <download arch="x86_64" url="http://ftp.de.debian.org/debian/jessie/main" repotype="deb">
      <pubkey>
-----BEGIN PGP PUBLIC KEY BLOCK-----
Version: GnuPG v1.4.12 (GNU/Linux)

...

      </pubkey>
    </download>
    <download arch="i586" url="http://ftp.de.debian.org/debian/jessie/main" repotype="deb">
      <pubkey>
-----BEGIN PGP PUBLIC KEY BLOCK-----
Version: GnuPG v1.4.12 (GNU/Linux)

...

      </pubkey>
    </download>
    <arch>x86_64</arch>
    <arch>i586</arch>
  </repository>
</project>

3.4.7.4 Arch Repository (arch)

Please be aware that there is no way atm to verify the origin of repository for Arch.

Example:

<project name="Arch:Core">
  
  ...
  
  <repository name="standard">
    <download arch="x86_64" url="http://ftp5.gwdg.de/pub/linux/archlinux/core/os/x86_64" repotype="arch"/>
    <download arch="i586" url="http://ftp5.gwdg.de/pub/linux/archlinux/core/os/i686" repotype="arch"/>
    <arch>x86_64</arch>
    <arch>i586</arch>
  </repository>
</project>

3.4.7.5 Mandriva Repository (mdk)

Example:

<project name="Mageia:5">

  ...

  <repository name="standard">
    <download arch="x86_64" url="http://mirror.example.org/Mageia/distrib/5/x86_64/media/core/release" repotype="mdk"/>
    <download arch="i586" url="http://mirror.example.org/mirrors/Mageia/distrib/5/i586/media/core/release" repotype="mdk"/>
    <arch>x86_64</arch>
    <arch>i586</arch>
  </repository>
</project>

3.5 Integrate External Source Repositories

3.5.1 Motivation

This chapter makes some recommendations how upstream resources can be integrated into the build process. SCM stands for source control management. git, subversion or CVS are concrete implementations of a SCM. The OBS itself comes also with an own SCM, but this is only intended to manage the files needed for packaging. However, you can add references to external SCM systems. The source service system will mirror the sources and provide it to the build systems. OBS makes sure that you can access the sources of all builds also in the future, even when the upstream server delivers different or no content at all anymore. Using external SCM references have the following advantages

  • It is documented where a source comes from and how to create the archive.
  • Working on the upstream sources can be done directly in local checkouts and changes can be tested via local builds before pushing to the SCM server.
  • The sources can be stored incremental and need less storage on the server.

3.5.2 Creating a external SCM reference

External refernces are defined in _service files. The file can look like this:

<services>
  <service name="obs_scm">
    <param name="url">git://...</param>
    <param name="scm">git</param>
  </service>
  <service name="tar" mode="buildtime"/>
  <service name="recompress" mode="buildtime">
    <param name="file">*.tar</param>
    <param name="compression">xz</param>
  </service>
  <service name="set_version" mode="buildtime" />
</services>

The services are doing the following:

  • obs_scm: is mirroring the source. It is storing it as cpio archive, but for the build process this looks like a directory. It stores also additional information from the meta data to a file with obsinfo suffix.
  • tar: creates a tar ball from the directory
  • recompress: applies a compression on the tar ball
  • set_version: reads the version from the obsinfo file and adapts the build descriptions to it.

Please note that only the first service (obs_scm) is running on the OBS server. The other services are running during the build process. They can also be replaced by any user by providing alternative implementations of them or writing their own service from scratch.

3.5.3 Working with local checkouts

Using

            osc build

in any package with such a defintion will do the same process local. The only difference is that you get a local subdirectory with the SCM content. You can go inside and work as you are used to. Any changes inside will be used for your next local build. No matter if they got pushed to the upstream server or not. However, you need to push it to upstream when you let the OBS server re-fetch the changes from upstream. The only way out would be to set the "obs_scm" service to mode disabled and upload your local archive.

3.5.4 Manage your build description in SCM

The "obs_scm" service allows you to export files next to the archive. You can specify one or more files using the "extract" parameter. Use it for your build description files.

3.6 Attribute System

...

3.7 Automatic source processing

...

Print this page