Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 1687944 - keepalived container fails to restart during minor upgrade, hanging docker and stalling upgrade. [NEEDINFO]
Summary: keepalived container fails to restart during minor upgrade, hanging docker an...
Keywords:
Status: ASSIGNED
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-containers
Version: 14.0 (Rocky)
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: ---
Assignee: Jose Luis Franco
QA Contact: Marius Cornea
Andrew Burden
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-03-12 17:43 UTC by Alistair Tonner
Modified: 2019-03-26 04:37 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:
Target Upstream Version:
jfrancoa: needinfo? (atonner)


Attachments (Terms of Use)

Description Alistair Tonner 2019-03-12 17:43:11 UTC
Description of problem:
 During minor update of OSP 14 on RHEL7.6 during pre-upgrade tasks keepalived goes down and does not restart.  Docker hangs.  Upgrade stalls and times out.


Version-Release number of selected component (if applicable):
Installed repositories (rhel-7.6):
  14
  ceph-3
  ceph-osd-3
  rhel-7.6
docker-1.13.1-91.git07f3374.el7.x86_64
containerd version:  (expected: aa8187dbd3b7ad67d8e5e3a15115d3eef43a7ed1)
runc version: 290a33602b16ff2d1cc5339bc0297f0e094462ce (expected: 9df8b306d01f59d3a8029be411de015b7304dd8f)
init version: fec3683b971d9c3ef73f284f176672c44b448662 (expected: 949e6facb77383876aeff8a6944dde66b3089574)

192.168.24.1:8787/rhosp14/openstack-keepalived                   2019-03-06.1        d2f2329685cf        8 days ago          535 MB



How reproducible:

Build RHOS14 undercloud:1,controller:3,compute:1,ceph:3,ironic:2 and run a minor update against the stack.


Steps to Reproduce:

Actual results:

ar 12 10:58:44 undercloud-0.redhat.local dockerd-current[516710]: Unexpected argument(s): | tee /var/log/keepalived.log
Mar 12 10:58:44 undercloud-0.redhat.local dockerd-current[516710]: Stopped Keepalived v1.3.5 (03/19,2017), git commit v1.3.5-6-g6fa32f2
Mar 12 10:58:44 undercloud-0.redhat.local dockerd-current[516710]: Stopped
Mar 12 10:58:44 undercloud-0.redhat.local Keepalived[38986]: Stopped Keepalived v1.3.5 (03/19,2017), git commit v1.3.5-6-g6fa32f2
Mar 12 10:58:44 undercloud-0.redhat.local dockerd-current[516710]: Unexpected argument(s): | tee /var/log/keepalived.log
Mar 12 10:58:44 undercloud-0.redhat.local Keepalived_vrrp[39033]: Stopped
Mar 12 10:58:44 undercloud-0.redhat.local dockerd-current[516710]: time="2019-03-12T10:58:44.164609694-04:00" level=debug msg="attach: stderr: begin"
 


Expected results:

Update completes normally

Additional info:

Comment 1 Jose Luis Franco 2019-03-18 14:13:13 UTC
Having a look at the docker version this looks to me very similar to the issue described in https://bugzilla.redhat.com/show_bug.cgi?id=1671861 . Could we have more logs please? Was there a  restart of the docker service? if that could be confirmed then we can ensure this is a duplicate of 1671861.

Comment 2 Jose Luis Franco 2019-03-18 14:14:44 UTC
A good way to verify if we're facing the same issue is to login in some of the overcloud controller nodes and perform a "docker ps", if the command hangs then it's exactly the same issue described in 1671861.


Note You need to log in before you can comment on or make changes to this bug.