Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 1364949 - jenkinspipeline builds skip running status in openshift client after restart master api server
Summary: jenkinspipeline builds skip running status in openshift client after restart ...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Image
Version: 3.3.0
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: ---
: ---
Assignee: Jimmi Dyson
QA Contact: Wang Haoran
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-08-08 09:31 UTC by XiuJuan Wang
Modified: 2017-03-08 18:43 UTC (History)
7 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: API server restart causes the Jenkins sync plugin to lose its connection to OpenShift Consequence: Pipeline builds are not properly executed in the Jenkins server Fix: Update sync plugin to handle connection loss when API server is restarted Result: Builds will be properly processed by the sync plugin and executed in Jenkins if the API server is restarted
Clone Of:
Environment:
Last Closed: 2017-01-18 12:52:08 UTC


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2017:0066 normal SHIPPED_LIVE Red Hat OpenShift Container Platform 3.4 RPM Release Advisory 2017-01-18 17:23:26 UTC

Description XiuJuan Wang 2016-08-08 09:31:58 UTC
Description of problem:
After restarting master api server, new triggered builds will become to complete/failed/cancelled directly from new status, no running staus in openshift client.

Works well before restart master.

Version-Release number of selected component (if applicable):
ci.dev.../openshift/jenkins-slave-nodejs-rhel7 965eb5e5766a
ci.dev.../openshift/jenkins-slave-maven-rhel7  7c6bbfb9656e
ci.dev.../openshift/jenkins-1-rhel7 bd3ab5d39d48

openshift v3.3.0.16
kubernetes v1.3.0+507d3a7

How reproducible:
always

Steps to Reproduce:
1.Create a jenkins pipeline app.
2.Trigger some builds from openshift client and jenkins webconsole
3.Restart master api server
4.Trigger some builds from openshift client and jenkins webconsole again.
5.Check builds from openshift client and jenkins webconsole

Actual results:
jenkinspipeline builds skip running status in openshift client

Expected results:
Should sync all build status between openshift client and jenkins webconsole.

Additional info:

Comment 1 Ben Parees 2016-08-16 13:35:24 UTC
also tracking as: https://github.com/fabric8io/openshift-jenkins-sync-plugin/issues/99

Comment 2 Jimmi Dyson 2016-09-21 14:14:49 UTC
This is a problem if the API server host is rebooted. If the API server process is stopped/started then it reconnects fine. This is a TCP timeout issue that affects oc client too (try it with `oc get pods -w` & then force reboot the API server - notice no timeout). I'm looking at implementing WS pings but not sure if API server supports pings (it should if it's spec compliant).

Comment 3 Jimmi Dyson 2016-09-21 15:56:27 UTC
This is a fix in kubernetes-client - PR https://github.com/fabric8io/kubernetes-client/pull/517. Will release client after merged & update jenkins plugin with new version after that.

Comment 4 Jimmi Dyson 2016-09-23 14:42:10 UTC
I've released openshift-jenkins-sync-plugin 0.0.13 (https://repo.jenkins-ci.org/releases/io/fabric8/jenkins/plugins/openshift-sync/0.0.13/) which should fix this issue.

@bparees: Forgotten how to get the Jenkins image updated with this new version?

Comment 5 Ben Parees 2016-10-06 21:47:44 UTC
centos image is available w/ the updated plugin.  rhel has not been rebuilt yet.

Comment 6 Troy Dawson 2016-11-02 18:05:14 UTC
New jenkins-1 and jenkins-2 images have been built with the updated openshift-sync plugin.

Comment 8 Dongbo Yan 2016-11-03 06:51:28 UTC
Verified
brew-pulp.../openshift3/jenkins-1-rhel7  6f476676a4a1
openshift-sync	0.0.14

openshift v3.4.0.19+346a31d
kubernetes v1.4.0+776c994
etcd 3.1.0-rc.0

Actual result:
After restart master api,new triggered pipeline builds could be running status, then change to complete.

Comment 10 errata-xmlrpc 2017-01-18 12:52:08 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:0066


Note You need to log in before you can comment on or make changes to this bug.