Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 453000 - Rgmanager stops heartbeat interface is Virtual IP is substring match to the broadcast address
Summary: Rgmanager stops heartbeat interface is Virtual IP is substring match to the b...
Alias: None
Product: Red Hat Enterprise Linux 5
Classification: Red Hat
Component: rgmanager
Version: 5.2.z
Hardware: All
OS: Linux
Target Milestone: rc
: ---
Assignee: Lon Hohberger
QA Contact: Cluster QE
Depends On:
TreeView+ depends on / blocked
Reported: 2008-06-26 14:48 UTC by Shane Bradley
Modified: 2018-10-20 03:08 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Last Closed: 2009-01-20 20:56:32 UTC
Target Upstream Version:

Attachments (Terms of Use)
rgmanager mismatch ip (deleted)
2008-06-26 15:02 UTC, Shane Bradley
no flags Details | Diff

System ID Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2009:0101 normal SHIPPED_LIVE rgmanager bug-fix and enhancements update 2009-01-20 16:04:34 UTC

Description Shane Bradley 2008-06-26 14:48:35 UTC
From Bugzilla Helper:
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9) Gecko/2008061712 Fedora/3.0-1.fc9 Firefox/3.0

Description of problem:

Problem description: On a cluster node where there is an rgmanager IP resource that is a substring match to the broadcast address of any interface, stopping or relocating that service will bring down that entire interface.  If this happens to be the interface used for cluster communication then the node misses its heartbeats and gets fenced.

Example: ashprdgfs01 has eth0 =  This means the broadcast address is  Stopping the following resource:

  <ip address="" monitor_link="1"/>

causes the wrong ip address to be removed:

  Jun 23 15:18:27 ashprdgfs01 clurgmgrd[3889]: <notice> Stopping service service:VIP
  Jun 23 15:18:27 ashprdgfs01 clurgmgrd: [3889]: <info> Removing IPv4 address from eth0

Since this is the main ip for eth0, this kills cluster communication and the node gets fenced:

  Jun 23 15:18:42 ashprdgfs02 kernel: dlm: closing connection to node 2
  Jun 23 15:18:42 ashprdgfs02 fenced[3199]: fencing node ""

The problem is on line 714 of

  addr=`/sbin/ip addr list | grep "$addr" | head -n 1 | awk '{print $2}'`

Because is a substring of, this actually returns 2 lines:

   inet brd scope global eth0
   inet scope global secondary eth0

and the first one is chosen mistakenly, causing it to be the one removed (718):

   /sbin/ip -f inet addr del dev $dev $addr

The address we want will always be follwed by a '/' for the subnet, so this can be fixed easily like so:

   addr=`/sbin/ip addr list | grep "$addr/" | head -n 1 | awk '{print $2}'`

Version-Release number of selected component (if applicable):

How reproducible:

Steps to Reproduce:
1) Configure an interface with a netmask
2) Configure an IP resource on that interface with last octet of .25 or .2
3) Stop the service:

     # clusvcadm -s vip

Actual Results:
VIP service should stop, leaving the main ip on that interface intact. If this is the heartbeat interface then the node loses communication with cluster and node is fenced off.

Expected Results:
The entire interface is taken down.

Additional info:
Workaround is to not use VIP that is substring of broadcast.

Comment 1 Shane Bradley 2008-06-26 15:02:20 UTC
Created attachment 310344 [details]
rgmanager mismatch ip

Comment 2 Lon Hohberger 2008-06-26 17:58:06 UTC

On RHEL4, that line isn't even present.

Comment 3 RHEL Product and Program Management 2008-07-03 18:21:38 UTC
This request was evaluated by Red Hat Product Management for inclusion in a Red
Hat Enterprise Linux maintenance release.  Product Management has requested
further review of this request by Red Hat Engineering, for potential
inclusion in a Red Hat Enterprise Linux Update release for currently deployed
products.  This request is not yet committed for inclusion in an Update

Comment 4 Fedora Update System 2008-07-30 20:03:55 UTC
gfs2-utils-2.03.05-1.fc9, rgmanager-2.03.05-1.fc9, cman-2.03.05-1.fc9 has been pushed to the Fedora 9 stable repository.  If problems still persist, please make note of it in this bug report.

Comment 5 Lon Hohberger 2008-08-01 16:13:36 UTC
This breaks when netmasks are used.

Comment 6 Lon Hohberger 2008-08-01 16:16:29 UTC
@@ -702,7 +702,7 @@
 			if [ "${addr/\/*/}" != "$ifaddr" ]; then
-			addr=`/sbin/ip addr list | grep "$addr" | head -n 1 | awk '{print $2}'`
+			addr=`/sbin/ip addr list | grep "$addr/" | head -n 1 | awk '{print $2}'`

... should be:

+			addr=`/sbin/ip addr list | grep "$ifaddr/" | head -n 1 | awk '{print $2}'`

Comment 7 Lon Hohberger 2008-08-22 18:04:24 UTC;a=commit;h=40e34ea406411c10a53aa1a147a0c94a62c3d12b

Ancillary patch has been pushed to RHEL5 branch.

Comment 8 Lon Hohberger 2008-08-22 18:07:06 UTC
Pushed to STABLE2 and master branches as well.

Comment 9 Lon Hohberger 2008-09-03 15:31:15 UTC
*** Bug 460541 has been marked as a duplicate of this bug. ***

Comment 12 errata-xmlrpc 2009-01-20 20:56:32 UTC
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

Note You need to log in before you can comment on or make changes to this bug.