3 Security Concepts #
3.1 General Paradigm #
The general paradigm of Open Build Service is to host all content on its own. Every part required to rebuild a package is hosted in Open Build Service to guarantee reproducibility. This includes the ability to rebuild the build environment. However, optional services to integrate remote resources exist as well. These resources are either mirrored and stored in revision control system or just cached.
3.1.1 Frontend #
The API and web interface frontends is the only part which must be accessible from public network. A SSL/TLS certificate is highly recommended.
3.1.1.1 Access to Mirror Servers #
The following services require access to stage servers. These servers can be used to publish content without the need to make Open Build Service server parts public available.
The publisher is used to publish any build results to the stage servers. It needs access for rsync service to the servers.
The source publisher can be used similar to publish sources belonging to published build results (optional service).
3.1.1.2 Access to the Public Network #
The following services may require access to the public network.
The srcserver when using the interconnect functionality to use content from remote Open Build Service instances.
obsdodup when using external repositories (optional service). The obssrcserver would need access then as well for downloading content.
obssourceservice when supporting automatic source procession (optional service) from remote. This service may be used to download content (for example, Git repositories) as part of the sources.
obsclouduploadworker when publishing to public cloud instances is wanted.
3.1.1.3 Worker network #
It is recommended to run the Open Build Service workers in an isolated network. This is an additional security mechanism in case of a security breach on a worker. This network needs access to the source and repository servers of the Open Build Service backend, but nowhere else.
3.1.1.4 Signer network #
It is recommended to run the signd on an isolated host. The signer services need to stay on the Open Build Service backend servers, they are just used for scheduling sign jobs. The signd is the critical instance which hosts the single instance signature key. All further keys which belong to any project are created by the signd, crypted with the instance key and delivered to the backend servers. That way the signd instance is stateless after initial setup and it is enough to backup the backend servers. Any sign job sends the encrypted private key to the signd server which is decrypting it for signing the content. Access to the signd server must be limited to Open Build Service backend server components (source server and publisher).
3.1.2 Build Environment #
The build environment is created by obsworker instances via the build script. Inside the build instances unverified and potentially harmful code is executed. Given that a user can run also any provided kernel the isolation must happen on VM layer. The only supported VM types, which are considered as secure are KVM and XEN VMs. The VMs prohibit any network access from the running instances. The build script is always creating a new file system, copies in all required data and executes the VM.
The build results get extracted directly from the block device. This is done to avoid mounting it which could be used to breach the host kernel in the file system code by crafting the used filesystem during build.
Every build also stores the used binaries inside of the _buildenv file. This
can be used to re-create the build environment later, even when newer
updates got released meanwhile. This requires however a maintenance setup
to avoid that former releases get removed.
The build tools (like rpmbuild) are running usually as non-root user, but this is not a security instrument. It is only a quality mechanic to ensure that source packages are rebuildable without root permissions.
3.1.3 Source Revision System #
The source revision storage system is part of Open Build Service. The identification of sources still happens using MD5 sums for historic reasons. MD5 is considered to be still good enough for identifying a source, but it is known to be attackable. Recent versions of the osc command line tooling is sending therefore also a SHA256 sum in addition which is used to detect collisions by the source server.
Sources can reference other package sources via _link files. These can be pinned
to a revision or to always using the latest one. That way underlying changes
get merged automatically. This is useful for shared development and automated
builds, but it should be avoided for base projects.
3.1.4 Permission Handling #
Authorization for write operations is done via the maintainer role on package or project level. On project level the projects are organized in namespaces which are defined via colons inside the project name. A maintainer role on a higher namespace grants the permission to claim maintainer role on any deeper project. Official projects should therefore be organized under their own top level namespaces (for example, openSUSE: namespace in our reference instance).
3.1.5 Signature Handling #
Signatures are used to proof the origin of a shipment independent of
Open Build Service instance. Once the signd daemon has
been enabled by the administrator, any binary result like RPM
packages, images, containers, or meta data gets signed.
All Open Build Service instances have a master key to sign results. This master
key has two purposes:
First, it is used for signing if a dedicated signing key
can not be found. Second, it is used to encrypt dedicated
project signing keys so that they can be stored inside of the
projects instead of on the signing server.
If a project does not have a signing key, the build server will
search the parent projects. This follows the logic of the write
permission handling.
An Open Build Service user (including the Open Build Service admin user) has only limited options to deal with the key setup. The user can create, delete, or extend the expire time of a key. It is not possible to import an external key. This guarantees that no key can be used outside of the Open Build Service instance to sign Open Build Service content.
A SSL certificate is derived from the keys when needed (for example, for secure boot).
GPG keypairs are created by the signd daemon. Therefore,
it is recommend to run this daemon on a separated and protected host. The
master keypair should exist only on this signd host. Any further
created keypair is not stored on the signd instance, instead
the private part is encrypted with the master key and transferred
to the Open Build Service backend. Thus, the signd instance is stateless and
needs no recurring backups after initial setup.
All keypairs (public and private parts) are therefore part of the
backup of the Open Build Service backend servers. The sign executable
transfers just the hash to be signed (not the entire file content) together
with the encrypted private key to signd. The daemon
decrypts the private key with the master key, creates the signature, and sends it
back to the client. The returned signature is applied to the binary by the sign
executable afterwards.
A compromised backend would still result in a serious security incident since any content can be signed with any project key. The private keys are not compromised themselves though.
3.2 Trust Zones #
Open Build Service (OBS) components deal with different trust zones. These are separated via network or virtualization mechanics.
3.2.1 Public Zones #
Public zones are areas where any code under user control is running.
3.2.1.1 External Network #
This can be the public Internet if the Open Build Service instance is a public instance. Requests can only be triggered via http, secured by SSL to the API instance here. Every developer is using this interface to submit changes. However, further components may open connections to the Internet as described below.
3.2.1.2 Untrusted Code #
All code which is used to build content is considered to be untrusted code. This includes even the Linux kernel, since users can build and run their own kernels. The security layer here is usually either KVM and XEN virtualization. The instances are running without any network enabled. Build results get copied by reading a simple blocklist directly from the device. A security issue in the file system code can therefore not be used to compromise the worker.
3.2.2 Demilitarized Zone (DMZ) #
The Demilitarized Zone contains services which interact with the public zone directly.
3.2.2.1 Open Build Service Frontend #
The frontend service is the only service which provides an open port. It implements the authorization of user requests. The authentication may happen in an external service like a proxy or ldap server. The frontend reads and writes changes to the database and the source server only.
3.2.2.2 Open Build Service Frontend Background Services #
Open Build Service frontend background services handle less time critical operations. This includes services which read data from external services like bug trackers, sending notifications or long running jobs.
3.2.2.3 Stage Server #
The stage server is providing the public content of the Open Build Service backends. The server can be publicly accessible or just an upload server to a mirror infrastructure.
3.2.2.4 Cloud Uploader #
The cloud uploader is uploading build results on user request. It reads from the binary servers and sends the content via cloud specific plugins to external instances. This is an optional service.
3.2.2.5 Source Service Server #
The source service server is acting based on uploaded sources. The services should be written with security in mind and the administrator can decide which services are trustworthy. Official releases of Open Build Service define a minimum set of especially trusted services which received a security review. A container can be used for additional security for each service run.
3.2.3 Internal Zone #
The internal zone is running service which are supposed to work without further external dependency.
3.2.3.1 Open Build Service Source Server #
The source server coordinates changes to package and project configuration. In addition to that all events between the binary backends and notifications to the frontend get synchronized. Any source change is tracked and stored in the revision history. There can only be a single source server per OBS install.
3.2.3.2 Open Build Service Binary Servers #
Binary Servers are hosting all content of build results. They also prepare public repositories and deliver them to the staging server.
3.2.3.3 External Dependencies #
The internal zone has no external dependency. However, the administrator may decide to run the Open Build Service instance depending on external OBS instances or on external repositories. In that case the internal zone is downloading content from the resources specified by the administrator.
3.2.4 Worker Zone #
The Open Build Service workers are running in an own isolated network. They access only source and binary servers from internal zone.
