Jump to contentJump to page navigation: previous page [access key p]/next page [access key n]
User Guide / Reference / OBS Concepts

22 OBS Concepts

We describe here the high-level concepts: how Open Build Service is designed, manages its content and is supposed to work.

22.1 Project Organization

All sources and binaries which are hosted inside of OBS are organized into projects. A project is the container defining a larger task. It defines who is working there.

22.1.1 Project Metadata

A project is configured in the project /source/$PROJECT/_meta path. It can be edited in the web interface using the RAW Config tab or via command line with

osc meta prj -e $PROJECT

This file contains:

  • Generic description data in title and description elements.

  • An ACL list of users and groups connected with a role. The maintainer role defines the list of users permitted to commit changes to the project.

  • A number of flags controlling the build and publishing process and possible read access protections.

  • A list of repositories to be created. This list defines what other repositories should be used, which architectures shall be built and build job scheduling parameters.

The following flags can be used to control the behavior of a package or project. Most of them can also be limited to specified repositories or architectures.

  • build defines whether package sources should get built. If enabled, it signals the scheduler to trigger server-side builds based on events like source changes, changes of packages used in the build environment or manual rebuild triggers. A local build via CLI is possible independent of this flag. Default is enabled.

  • publish can be used to enable or disable publishing the build result as repository. This happens after an entire repository has finished building for an architecture. A publish also gets triggered when the publish flag is enabled after a repository finishes the build. Default is enabled.

  • debuginfo can be used to modify the build process to create debuginfo data along with the package build for later debugging purposes. Changing this flag does not trigger rebuilds, it just affects the next build. Default is disabled.

  • useforbuild is used to control if a built result shall be copied to the build pool. This means it will get used for other builds in their build environment. When this is disabled, the build has no influence on builds of other packages using this repository. In case a previous build exists the old binaries will be used. Disabling this flag also means that "wipe" commands to remove binary files will have no effect on the build pool. Changing this flag does not trigger rebuilds, it just affects the next build. Default is enabled.

  • access flag can be used to hide an entire project. This includes binaries and sources. It can only be used at project creation time and can just be enabled (making it public again) afterwards. This flag can only be used on projects. Default is enabled.

  • sourceaccess flag can be used to hide the sources, but still show the existence of a project or package. This also includes debug packages in case the distribution is supporting this correctly. This flag can only be used at package creation time. There is no code yet which checks for possible references to this package. Default is enabled.

  • downloadbinary permission still exists like before. However, unlike "access" and "sourceaccess" this is not a security feature. It is just a convenience feature, which makes it impossible to get the binaries via the API directly. But it is still possible to get the binaries via build time in any case. Default is enabled.

22.1.2 Project Build Configuration

A project is configured in the project /source/$PROJECT/_config path. It can be edited in web interface in the Project Config tab or via one of the following command lines

osc meta prjconf -e $PROJECT
osc co $PROJECT _project

The build configuration is used to tell the OBS how to set up the build environment, and is explained in Chapter 4, Build Configuration.

22.1.3 Project Build Macro Configuration

The macro configuration is part of the build configuration in /source/$PROJECT/_config. It can be added at the end after a Macros: line.

The build configuration is used to tell the OBS how to set up the build environment, and is explained in Chapter 4, Build Configuration. Within that chapter, build macros are explained in Section 4.4, “Macro Definitions in the Build Configuration”.

22.1.4 An OBS Package

An OBS Package is a sub-namespace below a project. It contains the specification of a single package build for all specified repositories.

22.2 The OBS Interconnect

The OBS interconnect is a mechanism to connect two OBS instances. All content, including sources and binary build results, will be available in the connecting instance. Unlike other methods the instances will also notify each other about changes.

22.3 Download on Demand Repositories (DoD)

22.3.1 Motivation

In a DoD repository external software repositories can be configured which are used for dependency resolution and where packages will be downloaded at build time. A DoD repository has some main advantages in comparison to binary import projects:

  • less disk usage as only really required packages will be downloaded

  • automatic package updates when new upstream releases are available

  • simple to configure in project meta with no for shell access to repo servers

In download repotypes where package checksums can be verified (e.g. susetags, rpmmd and deb), we recommend that you use a mirror server URL in <download> in order to reduce traffic on the master server and configure a <master> with an https url and a sslfingerprint in order to avoid man in the middle attacks by peer verification.

22.3.2 XML Document Hierarchy

<project>
  <repository>
    <download>
      <master/> (optional)
      <pubkey/> (optional)
    </download>
  </repository>
</project>

22.3.3 The Daemon

The bs_dodup daemon periodically checks for new metadata in remote repositories. This daemon can be enabled for startup with the command

systemctl enable obsdodup.service

and can be started with

systemctl start obsdodup.service

22.3.4 The download Element

mandatory attributes:

  • arch

  • url

  • repotype

22.3.5 The master Subelement

The <master> tag as shown in the rpmmd example below is optional but strongly recommended for security reasons.

Verification is supported in the following repotypes

  • susetags

  • rpmmd

  • deb

This option could be defined by any valid URL (HTTP and HTTPS) to the origin of the repository but it is strongly recommended to use https with a sslfingerprint to bs_dodup possibility to verify its peer in order to avoid man-in-the-middle attacks. The download URL can be a mirror as we validate package checksums found in repo data.

You can easily query the SSL fingerprint of a remote server with the following command:

openssl s_client -connect <host>:<port> < /dev/null 2>/dev/null | openssl x509 -fingerprint -noout

22.3.6 The pubkey Subelement

The pubkey element contains one or more GPG public keys in order to verify repository information but not packages. For an example, look at the repotype "deb" documentation below.

22.3.7 Repository Types

22.3.7.1 YAST Sources (susetags)

Example:

<project name="My::SuSE::CD">

  [...]

  <repository name="standard">
    <download arch="x86_64" url="http://mirror.example.org/path/to/iso" repotype="susetags" />
    <download arch="i586" url="http://mirror.example.org/path/to/iso" repotype="susetags" />
    <arch>x86_64</arch>
    <arch>i586</arch>
  </repository>
</project>

22.3.7.2 RPM Sources (rpmmd)

Example:

<project name="Fedora:Rawhide">

  [...]

  <repository name="standard">
    <download arch="x86_64" url="http://mirror.example.org/fedora/rawhide/x86_64/os" repotype="rpmmd">
      <master url="https://master.example.org/whereever/fedora/rawhide/x86_64/os" sslfingerprint="sha256:0a64..0303"/>
    </download>
    <download arch="i586" url="http://mirror.example.org/fedora/rawhide/i386/os" repotype="rpmmd">
      <master url="https://master.example.org/whereever/fedora/rawhide/i386/os" sslfingerprint="sha256:0a64..0303"/>
    </download>
    <arch>x86_64</arch>
    <arch>i586</arch>
  </repository>
</project>

22.3.7.3 Apt Repository (deb)

Apt supports two repository types, flat repositories and distribution repositories.

The download url syntax for them is:

  • <baseurl>/<distribution>/<components>

  • <flat_url>/.[/<components>]

You can specify multiple components separated by a comma.

An empty components string is parsed as "main".

Example:

<project name="Debian:8">

  [...]

  <repository name="ga">
    <download arch="x86_64" url="http://ftp.de.debian.org/debian/jessie/main" repotype="deb">
      <pubkey>
-----BEGIN PGP PUBLIC KEY BLOCK-----
Version: GnuPG v1.4.12 (GNU/Linux)

[...]

      </pubkey>
    </download>
    <download arch="i586" url="http://ftp.de.debian.org/debian/jessie/main" repotype="deb">
      <pubkey>
-----BEGIN PGP PUBLIC KEY BLOCK-----
Version: GnuPG v1.4.12 (GNU/Linux)

[...]

      </pubkey>
    </download>
    <arch>x86_64</arch>
    <arch>i586</arch>
  </repository>
</project>

22.3.7.4 Arch Repository (arch)

Be aware that there is currently no way to verify the origin of repository for Arch.

Example:

<project name="Arch:Core">

  [...]

  <repository name="standard">
    <download arch="x86_64" url="http://ftp5.gwdg.de/pub/linux/archlinux/core/os/x86_64" repotype="arch"/>
    <download arch="i586" url="http://ftp5.gwdg.de/pub/linux/archlinux/core/os/i686" repotype="arch"/>
    <arch>x86_64</arch>
    <arch>i586</arch>
  </repository>
</project>

22.3.7.5 Mandriva Repository (mdk)

Example:

<project name="Mageia:5">

  [...]

  <repository name="standard">
    <download arch="x86_64" url="http://mirror.example.org/Mageia/distrib/5/x86_64/media/core/release" repotype="mdk"/>
    <download arch="i586" url="http://mirror.example.org/mirrors/Mageia/distrib/5/i586/media/core/release" repotype="mdk"/>
    <arch>x86_64</arch>
    <arch>i586</arch>
  </repository>
</project>

22.4 Integrating External Source Repositories

22.4.1 Motivation

This chapter makes some recommendations how upstream resources can be integrated into the build process. SCM stands for source control management. git, subversion or CVS are concrete implementations of an SCM. The OBS itself comes also with an own SCM, but this is only intended to manage the files needed for packaging. However, you can add references to external SCM systems. The source service system will mirror the sources and provide it to the build systems. OBS makes sure that you can access the sources of all builds also in the future, even when the upstream server delivers different or no content at all anymore. Using external SCM references has the following advantages:

  • It is documented where a source comes from and how to create the archive.

  • Working on the upstream sources can be done directly in local checkouts and changes can be tested via local builds before pushing to the SCM server.

  • The sources can be stored incrementally and need less storage on the server.

22.4.2 Creating an Reference to an External SCM

External references are defined in _service files. The file can look like this:

<services>
  <service name="obs_scm">
    <param name="url">git://...</param>
    <param name="scm">git</param>
  </service>
  <service name="tar" mode="buildtime"/>
  <service name="recompress" mode="buildtime">
    <param name="file">*.tar</param>
    <param name="compression">xz</param>
  </service>
  <service name="set_version" mode="buildtime" />
</services>

The services do the following:

  • obs_scm: mirrors the source. It stores it as a cpio archive, but for the build process this looks like a directory. It also stores additional information from the metadata to a file with obsinfo suffix.

  • tar: creates a tar file from the directory

  • recompress: applies a compression on the tar file

  • set_version: reads the version from the obsinfo file and adapts the build descriptions to it.

Note that only the first service (obs_scm) runs on the OBS server. The other services run during the build process. They can also be replaced by any user by providing alternative implementations of them, or by writing their own service from scratch.

22.4.3 Working with Local Checkouts

Using osc build in any package with such a definition will do the same process locally. The only difference is that you get a local subdirectory with the SCM content. You can go inside and work as you are used to. Any changes inside will be used for your next local build, whether they were pushed to the upstream server or not. However, you need to push it upstream when you let the OBS server re-fetch the changes from upstream. The only way out would be to set the obs_scm service to mode disabled and upload your local archive.

22.4.4 Managing Build Recipes in a SCM

The obs_scm service allows you to export files next to the archive. You can specify one or more files using the extract parameter. Use it for your build recipe files.