1 OBS Architecture

Abstract

1.1 Overview Graph

Open Build Service (OBS) is not a monolithic server; it consists of multiple daemons that fulfill different tasks (see Figure 1.1, “Simplified OBS Component Overview”).

Simplified OBS Component Overview
Figure 1.1: Simplified OBS Component Overview

The OBS Back-end manages the source files and build jobs of the OBS.

Source Server

Maintains the source repository and project/package configurations. It provides an HTTP interface, which is the only interface for the Front-end and It may forward requests to further back-end services.

The Source Server keeps track of all sources that are available for building. It takes care of file deduplication so that every source file is stored only once. This is done by keeping track of the MD5 hashes of the files in combination with the file names. All revisions of committed sources are stored and will not be deleted. This guarantees the ability to get the source for every delivered binary package.

Each OBS installation has one Source Server only. It maintains the "sources", "trees" and "projects" directories.

Repository Server

A repository server provides access to the binaries via an HTTP interface. It is used by the front-end via the source server only. Workers use the server for registration, requesting the needed binaries for the build jobs and storing the result. Notifications for schedulers are also created by repository servers. Each OBS installation has at least one repository server. A larger installation using partitioning has one on each partition.

Scheduler

A scheduler calculates the need for build jobs. It detects changes in sources, project configurations or in binaries used in the build environment. It is responsible for starting jobs in the right order and integrating the built binary packages. Each OBS installation has one scheduler per available architecture and partition. It maintains the cntent of the "build" directory.

Dispatcher

The dispatcher takes a job (created by the scheduler) and assigns it to a free worker. It also checks possible build constraints to verify that the worker qualifies for the job. It only notifies a worker about a job; the worker downloads the needed resources itself afterwards. Each OBS installation has one dispatcher per partition, one of which is the master dispatcher.

The dispatcher tries to assign jobs fairly between the project repositories. For this the dispatcher maintains a load per project repository (similar to the Unix system load) of used build time. The dispatcher assigned jobs to build clients from the repository with the lowest load (thereby increasing its load). It is possible to tweak this mechanism via dispatching priorities assigned to the repositories via the /build/dispatchprios API call or via the dispatch_adjust map in the BSConfig.pm configuration file. See the dispatch priorities in reference guide for more details.

Publisher

The publisher processes publish events from the scheduler for finished repositories. It merges the build result of all architectures into a defined directory structure, creates the needed metadata, and may sync it to a download server. It maintains the content of the "repos" directory on the back-end. Each OBS installation has one publisher per partition.

Signer

The signer handles signing events and calls an external tool to execute the signing. Each OBS installation usually has one signer per partition and also on the source server installation.

Source Service Server

The Source Service Server helps to automate processes for continuous integration. The server can call different services for different tasks. It can download sources from websites and version control systems such as subversion and git. Services can also include working on the source to extract spec-files from archives, repacking the archives or adjusting version numbers in spec files. It is also often used to enforce policies by running checks. A failed check will appear as broken source and blocks a package from building.

The Source Service Server is optional and currently only one Source Service Server is supported.

Download on Demand Updater (dodup) (OBS version 2.7 or later)

The download on demand updater monitors all external repositories which are defined as download on demand resources. It polls for changes in the metadata and re-downloads the metadata in case. The scheduler will be notified to recalculate the build jobs depending on these repositories afterwards. Each OBS installation can have one dodup service running on each partition.

Delta Store (OBS version 2.7 or later)

The delta store daemon maintains the deltas in the source storage. Multiple obscpio archives can be stored in one deltastore to avoid duplication on disk. This service calculates the delta and maintains the delta store. Each OBS installation can have one delta store process running next to the source server.

Worker

The workers register with the repository servers. They receive build jobs from the dispatcher. Afterwards they download sources from the source server and the needed binaries from the repository server(s). They build the package using the build script and send the result back to the repository server. A worker can run on the same host as the other services, but most OBS installations have dedicated hardware for the workers.

1.2 Communication Flow

The communication flow can be split into 3 parts:

  1. User-to-front-end communication

  2. Front-end-to-source-server communication

  3. communication between source server and other back-end components, in first place the repository servers.

The user is using the front-end (via tools like osc) to communicate with the Open Build Service. The front-end is providing a web interface and also an API. The front-end is implemented as Ruby on Rails application. All communication happens via the http protocol (normally encrypted so https is used.

The communication between the front-end and the back-end also use the http protocol, using the back-end source server as gateway to most of the other back-end components.

The figure Figure 1.2, “OBS Communication” shows the communication flow between the OBS components if a package source (e.g. a _service file) was updated.

If a source file of a package was updated, the new source file is uploaded with an HTTP PUT operation to the front-end. The front-end may ask for authentication (if not already done) and check the access rights for the user. If everything is OK, the new file will be send to the back-end Source server via an HTTP PUT request as well. The source server stores the changed source (under revision control) and check if for this change source service runs are needed. If yes the Source Service server is informed via an HTTP PUT request of the _service file to run the requested services.

The Source Service server runs the different source services, e.g download the requested revision from a GIT server. After running all services it delivers the final sources back to the Source server, which then store these under revision control as well.

The Source server then notifies the Schedulers (for each architecture) about the change of the package via an event.

The Scheduler recalculates the package and project state and if all build requires are solved, a build Job is created and put in the Job pool. The notified Dispatcher checks for free worker which met the build constraints for the job and send the job to the worker.

obs_communication.png
Figure 1.2: OBS Communication

The Worker downloads the sources from the Source server and all required binary packages from the Repo server. The package then will be built, the worker is monitored by the warden service to detect any worker crashes.

If the build was finished the build results (which are also contain build logs) the results are uploaded to the Scheduler again and if requested the Signer does sign the packages. The Scheduler recalculate the project status if here are more dependent packages to build. If not it informs the Publisher to publish the new build results and create an updated version of the repository. The Publisher will also request signing of the repository metadata.

Print this page