2 OBS Architecture

2.1 Overview graph

The OBS is not a monolithic server; it consists of multiple daemons that fulfill different tasks.

Simplified OBS Component Overview
Figure 2.1: Simplified OBS Component Overview

2.2 Backend Components

The OBS Backend is a collection of Perl applications that manage the source files and build jobs of the OBS.

2.2.1 Source server

Maintains the source repository and project/package configurations. It provides an HTTP interface, which is the only interface for the Frontend and It may forward requests to further backend services.

The Source Server keeps track of all sources that are available for building. It takes care of file deduplication so that every source file is stored just once. This is done by keeping track of the MD5 hashes of the files in combination with the filenames. All revisions of committed sources are stored and will not be deleted. This guarantees the ability to get the source for every delivered binary package.

Each OBS installation has one Source Server only. It maintains the "sources", "trees" and "projects" directories.

2.2.2 Repository server

A repository server provides access to the binaries via an HTTP interface. It is used by the frontend via the source server only. Workers use the server for registration, requesting the needed binaries for the build jobs and storing the result. Notifications for schedulers are also created by repository servers. Each OBS installation has at least one repository server. A larger installation using partitioning has one on each partition.

2.2.3 Scheduler

A scheduler calculates the need for build jobs. It detects changes in sources, project configurations or in binaries used in the build environment. It is responsible for starting jobs in the right order and integrating the built binary packages. Each OBS installation has one scheduler per available architecture and partition. It maintains the content of the "build" directory.

2.2.4 Dispatcher

The dispatcher takes a job (created by the scheduler) and assigns it to a free worker. It also checks possible build constraints to verify that the worker qualifies for the job. It only notifies a worker about a job; the worker downloads the needed resources itself afterwards. Each OBS installation has one dispatcher per partition, one of which is the master dispatcher.

The dispatcher tries to assign jobs fairly between the project repositories. For this the dispatcher maintains a load per project repository (similar to the Unix system load) of used build time. The dispatcher assigned jobs to build clients from the repository with the lowest load (thereby increasing its load). It is possible to tweak this mechanism via dispatching priorities assigned to the repositories via the /build/dispatchprios API call or via the dispatch_adjust map in the BSConfig.pm configuration file. See the dispatch priorities in reference guide for more details.

2.2.5 Publisher

The publisher processes publish events from the scheduler for finished repositories. It merges the build result of all architectures into a defined directory structure, creates the needed metadata, and may sync it to a download server. It maintains the content of the "repos" directory on the backend. Each OBS installation has one publisher per partition.

2.2.6 Signer

The signer handles signing events and calls an external tool to execute the signing. Each OBS installation usually has one signer per partition and also on the source server installation.

2.2.7 Source service server

The Source Service Server helps to automate processes for continuous integration. The server can call different services for different tasks. It can download sources from websites and version control systems such as subversion and git. Services can also include working on the source to extract spec-files from archives, repacking the archives or adjusting version numbers in spec files. It is also often used to enforce policies by running checks. A failed check will appear as broken source and blocks a package from building.

The Source Service Server is optional and currently only one Source Service Server is supported.

2.2.8 Warden

The warden monitors the workers and detects crashed or hanging workers. Their build jobs will be canceled and restarted on another host. Each OBS installation can have one Warden service running on each partition.

2.2.9 Download on Demand Updater (dodup)

Note
Note

Need OBS version 2.7 or later.

The download on demand updater monitors all external repositories which are defined as download on demand resources. It polls for changes in the metadata and re-downloads the metadata in case. The scheduler will be notified to recalculate the build jobs depending on these repositories afterwards. Each OBS installation can have one dodup service running on each partition.

2.2.10 Delta Store

Note
Note

Need OBS version 2.7 or later.

The delta store daemon maintains the deltas in the source storage. Multiple obscpio archives can be stored in one deltastore to avoid duplication on disk. This service calculates the delta and maintains the delta store. Each OBS installation can have one delta store process running next to the source server.

2.2.11 Worker

The workers register with the repository servers. They receive build jobs from the dispatcher. Afterwards they download sources from the source server and the needed binaries from the repository server(s). They build the package using the build script and send the result back to the repository server. A worker can run on the same host as the other services, but most OBS installations have dedicated hardware for the workers.

2.3 Communication flow

The communication flow can be split into 3 parts:

  1. User to frontend communication
  2. frontend to source server communication
  3. communication between source server and other backend components, in first place the repository servers.

The user is using the frontend (via tools like osc) to communicate with the Open Build Service. The frontent is providing a web interface and also an api. The frontend is implemented as Ruby on rails application. All communcation happens via the http protocol (normally encrypted so https is used.

The communication between the frontend and the backend also use the http protocol, using the backend source server as gateway to most of the other backend components.

The figure Figure 2.2, “OBS communication” shows the comunication flow between the OBS components if a a package source (e.g. a _service file) was updated.

If a source file of a package was updated, the new source file is uploaded with an HTTP PUT operation to the frontend. The frontend may ask for authentification (if not already done) and check the access rights for the user. If everything is OK, the new file will be send to the backend Source server via an HTTP PUT request as well. The source server stores the changed source (under revision control) and check if for this change source service runs are needed. If yes the Source Service server is informed via an HTTP PUT request of the _service file to run the requested services.

The Source Service server runs the different source services, e.g download the requested revision from a GIT server. After running all services it delivers the final sources back to the Source server, which then store these under revision control as well.

The Source server then notifies the Schedulers (for each architecture) about the change of the package via an event.

The Scheduler recalculates the package and project state and if all build requires are solved, a build Job is created and put in the Job pool. The notified Dispatcher checks for free worker which met the build contrains for the job and send the job to the worker.

obs_communication.png
obs_communication_legend.png
Figure 2.2: OBS communication

The Worker downloads the sources from the Source server and all required binary packages from the Repo server. The package then will be built, the worker is monitored by the warden service to detect any worker crashes.

If the build was finished the build results (which are also contain build logs) the results are uploaded to the Scheduler again and if requested the Signer does sign the packages. The Scheduler recalculate the project status if here are more dependent packages to build. If not it informs the Publisher to publish the new build results and create an updated version of the repository. Th Publisher will also request signing of the repository meta data.

Print this page