News about "deployments"

deployments

Post-mortem: Downtime on March 23, 2018

by OBS Team posted on 23rd Mar 2018

After today’s deployment we faced a downtime of our reference server. We want to give you some insight into what happened. What happend? At 11:32 we deployed a new version to our reference server build.opensuse.org. Right after the deployment the application was not reachable anymore and our exception tracker (Errbit 💋) started to receive a few TypeError: superclass mismatch for class BsRequestAction exceptions. The cause of the error was not very obvious. All HTTP requests...

Read more...
deployments

Post-mortem: Deployment on March 15, 2018

by OBS Team posted on 16th Mar 2018

During yesterday’s deployment we faced some issues. We want to give you some insight into what happened. What, How, When and Why At 15:10 we deployed and we realized that there were lots of exceptions on errbit with rpmlint_log because of a non-existent route. The removed (changed) route was used in some javascript so it was not evident how to fix this easily with a routing helper. When trying to revert the route change we...

Read more...
deployments

Post-mortem: Deployment on January 17th, 2018

by David Kang, Björn Geuken, Moisés Déniz posted on 17th Jan 2018

During today's deployment we faced some issues. We had to disable rabbitmq support in build.opensuse.org for some hours. What happened? The deployment This morning, at 10:25am, we deployed as usual. Some minutes after the deployment was done we noticed some notifications from errbit, our exception tracking tool. We looked at the tracked exception and the related backlog and started to debug the issue. We recently applied a larger amount of rubocop rules, a style checking...

Read more...
deployments

Post-mortem: Deployment on December 14, 2017

by Ana María Martínez Gómez posted on 15th Dec 2017

During yesterday's deployment we faced some issues. We had to monkey patch some fixes and we want to give you some insight into what happened. There were some critical PRs deployed yesterday morning: Refactor of Event module We refactored the whole Event module. We renamed things, created many classes and moved a lot of code. https://github.com/openSUSE/open-build-service/pull/4191 Move ProjectLogRotate to 1 event per job Creating one job per event instead of running a job every some...

Read more...
deployments

Post-mortem: Extended Deployment time on August 28, 2017

by Manuel Schnitzer posted on 28th Aug 2017

During deployment, we were facing some issues and build.opensuse.org was not accessible for a couple of minutes. This sucks and that's why we want to give you some insight in what happened. Problems/Timeline 2017-08-23 We updated Rails to 5.1.3 and dropped one of our initializers which is no longer needed. (See #3659) 2017-08-28 09:21 UTC – We installed the newest OBS packages from our Unstable project. During the installation process, our Apache server gets restarted....

Read more...
deployments

Post-mortem: Extended Deployment time on July 19, 2017

by Björn Geuken posted on 19th Jul 2017

We did it again! Yesterday, on 19th of July 2017, we had an extended deployment time because of an issue during the deployment. Though this time it "only" took 15 minutes;-) This sucks and that's why we want to give you some insight in what happened. Problems/Timeline 19-07-2017 12:37 UTC – We installed the newest OBS packages from our Unstable project and ran the migrations. 12:40 UTC – Installation and migration finished. We checked the...

Read more...
deployments

Post-mortem: Extended Deployment time on June 30, 2017

by the OBS Team posted on 4th Jul 2017

On June 30, 2017 we had an extended deployment time of roughly 45 minutes for our reference server because of a couple of problems with one of the data migrations. We implemented a new feature, user notifications via RSS, that included a migration of data in our database. This migration was broken, causing this deployment to go terribly wrong. The frontend team afterward met to do a post-mortem to identify the problems, solutions and possible...

Read more...
Powered by Github Pages, Semantic UI and Noun Project