Jump to contentJump to page navigation: previous page [access key p]/next page [access key n]
User Guide › Reference › OBS Architecture
ContentsContents
User Guide
  1. About this Guide
  2. I First Steps
    1. 1 Beginnerʼs Guide
  3. II Concepts
    1. 2 Supported Build Recipes and Package Formats
  4. III Setup
    1. 3 osc, the Command Line Tool
    2. 4 Build Configuration
  5. IV Usage
    1. 5 Basic OBS Workflow
    2. 6 Local Building
    3. 7 Using Source Services
    4. 8 SCM/CI Workflow Integration
    5. 9 Staging Workflow
    6. 10 Notifications
    7. 11 Moderation
  6. V Best Practices
    1. 12 Using the OBS Web UI
    2. 13 Basic Concepts and Work Styles
    3. 14 How to integrate external SCM sources
    4. 15 Publishing Upstream Binaries
    5. 16 Bootstrapping
    6. 17 osc Example Commands
    7. 18 Advanced Project Setups
    8. 19 Building Kernel Modules
    9. 20 Common Questions and Solutions
  7. VI Reference
    1. 21 OBS Architecture
    2. 22 OBS Concepts
    3. 23 Build Process
    4. 24 Build Containers
    5. 25 Source Management
    6. 26 SCM Bridge
    7. 27 Supported Formats
    8. 28 Request And Review System
    9. 29 Image Templates
    10. 30 Multiple Build Description File Handling
    11. 31 Maintenance Support
    12. 32 Binary Package Tracking
    13. 33 Administration
    14. 34 Scheduling and Dispatching
    15. 35 Build Constraints
    16. 36 Building Preinstall Images
    17. 37 Authorization
    18. 38 Quality Assurance(QA) Hooks
    19. 39 openSUSE Factory
  8. Glossary
  9. A GNU Licenses
Navigation←→
User Guide › Reference › OBS Architecture
Top
←→

21 OBS Architecture

Abstract

21.1 Overview Graph
21.2 Communication Flow

21.1 Overview Graph

Open Build Service (OBS) is not a monolithic server; it consists of multiple daemons that fulfill different tasks (see Figure 21.1, “Simplified OBS Component Overview”).

Simplified OBS Component Overview
Figure 21.1: Simplified OBS Component Overview

The OBS Back-end manages the source files and build jobs of the OBS.

Source Server

Maintains the source repository and project/package configurations. It provides an HTTP interface, which is the only interface for the Front-end and It may forward requests to further back-end services.

The Source Server keeps track of all sources that are available for building. It takes care of file deduplication so that every source file is stored only once. This is done by keeping track of the MD5 hashes of the files in combination with the file names. All revisions of committed sources are stored and will not be deleted. This guarantees the ability to get the source for every delivered binary package.

Each OBS installation has one Source Server only. It maintains the "sources", "trees" and "projects" directories.

Repository Server

A repository server provides access to the binaries via an HTTP interface. It is used by the front-end via the source server only. Workers use the server for registration, requesting the needed binaries for the build jobs and storing the result. Notifications for schedulers are also created by repository servers. Each OBS installation has at least one repository server. A larger installation using partitioning has one on each partition.

Scheduler

A scheduler calculates the need for build jobs. It detects changes in sources, project configurations or in binaries used in the build environment. It is responsible for starting jobs in the right order and integrating the built binary packages. Each OBS installation has one scheduler per available architecture and partition. It maintains the content of the "build" directory.

Dispatcher

The dispatcher takes a job (created by the scheduler) and assigns it to a free worker. It also checks possible build constraints to verify that the worker qualifies for the job. It only notifies a worker about a job; the worker downloads the needed resources itself afterwards. Each OBS installation has one dispatcher per partition, one of which is the master dispatcher.

The dispatcher tries to assign jobs fairly between the project repositories. For this the dispatcher maintains a load per project repository (similar to the Unix system load) of used build time. The dispatcher assigned jobs to build clients from the repository with the lowest load (thereby increasing its load). It is possible to tweak this mechanism via dispatching priorities assigned to the repositories via the /build/_dispatchprios API call or via the dispatch_adjust map in the BSConfig.pm configuration file. See the dispatch priorities in reference guide for more details.

Publisher

The publisher processes publish events from the scheduler for finished repositories. It merges the build result of all architectures into a defined directory structure, creates the needed metadata, and may sync it to a download server. It maintains the content of the "repos" directory on the back-end. Each OBS installation has one publisher per partition.

Signer

The signer handles signing events and calls an external tool to execute the signing. Each OBS installation usually has one signer per partition and also on the source server installation.

Source Service Server

The Source Service Server helps to automate processes for continuous integration. The server can call different services for different tasks. It can download sources from websites and version control systems such as subversion and git. Services can also include working on the source to extract spec-files from archives, repacking the archives or adjusting version numbers in spec files. It is also often used to enforce policies by running checks. A failed check will appear as broken source and blocks a package from building.

The Source Service Server is optional and currently only one Source Service Server is supported.

Download on Demand Updater (dodup) (OBS version 2.7 or later)

The download on demand updater monitors all external repositories which are defined as download on demand resources. It polls for changes in the metadata and re-downloads the metadata in case. The scheduler will be notified to recalculate the build jobs depending on these repositories afterwards. Each OBS installation can have one dodup service running on each partition.

Delta Store (OBS version 2.7 or later)

The delta store daemon maintains the deltas in the source storage. Multiple obscpio archives can be stored in one deltastore to avoid duplication on disk. This service calculates the delta and maintains the delta store. Each OBS installation can have one delta store process running next to the source server.

Worker

The workers register with the repository servers. They receive build jobs from the dispatcher. Afterwards they download sources from the source server and the needed binaries from the repository server(s). They build the package using the build script and send the result back to the repository server. A worker can run on the same host as the other services, but most OBS installations have dedicated hardware for the workers.

21.2 Communication Flow

The communication flow can be split into the following major parts:

  1. communication between users and front-end

  2. communication between front-end and source server

  3. communication between source server and other back-end components, in particular the repository servers.

  4. communication between the back-end and the stage server to publish build results

The user uses the front-end (via tools like osc) to communicate with the Open Build Service. The front-end is providing a web interface and also an API. The front-end is implemented as a Ruby on Rails application. All communication happens via the HTTP protocol (usually encrypted, meaning HTTPS is used).

The communication between the front-end and the back-end also uses the HTTP protocol, using the back-end source server as the gateway to most other back-end components.

OBS Communication (Simplified)
Figure 21.2: OBS Communication (Simplified)

The figure Figure 21.2, “OBS Communication (Simplified)” shows the communication flow between the OBS components if a package source (for example, a _service file) was updated:

  1. The new source file is uploaded with an HTTP PUT operation to the front-end. The front-end authenticates and checks the permissions of the user.

  2. If the user has appropriate permissions, the new file will be sent to the back-end source server via an HTTP PUT request. The source server stores the changed source under revision control.

    It then checks whether this change makes source service runs necessary. If so, the source service server is informed via an HTTP PUT request of the _service file to run the requested services.

  3. The source service server runs all required source services. For example, it could download the requested revision from a Git server. After running all services, it delivers the final sources back to the source server, which then stores these under revision control.

  4. The source server then notifies the schedulers for each hardware architecture required about the change of the package via an event.

  5. The scheduler then recalculates package and project state. If all build requirements are fulfilled, a build job is created and added to the job pool.

  6. The dispatcher is notified and selects a free worker which meets the build constraints for the job and sends the job to it.

  7. The worker downloads the sources from the source server and all required binary packages from the repository server. The package then will be built.

    The worker is monitored by the warden service to detect any worker crashes.

  8. If the build succeeds, the build results (including build logs) are uploaded to the scheduler. If requested, the signer signs the packages.

  9. The scheduler recalculates the project status, checking whether dependent packages need to be rebuilt. If not, it requests the publisher to publish the build results.

  10. The publisher will create an updated version of the output repository and request the signer to sign the repository metadata.

→Chapter 22 OBS Concepts←Part VI Reference
Print this page

© 2024 SUSE