Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 1053900 - [RFE] Hypervisor RHEV-H only connects to the Dell Equallogic using one connection. it is expected to do multipathing and have 4 connection to it.
Summary: [RFE] Hypervisor RHEV-H only connects to the Dell Equallogic using one connec...
Keywords:
Status: CLOSED DUPLICATE
Alias: None
Product: Red Hat Enterprise Virtualization Manager
Classification: Red Hat
Component: RFEs
Version: unspecified
Hardware: x86_64
OS: Linux
unspecified
high
Target Milestone: ---
: 3.4.0
Assignee: Sergey Gotliv
QA Contact: Leonid Natapov
URL:
Whiteboard: storage
: 953343 (view as bug list)
Depends On: 953343
Blocks: rhev3.4beta 1142926
TreeView+ depends on / blocked
 
Reported: 2014-01-15 23:10 UTC by Sean Cohen
Modified: 2014-09-18 12:24 UTC (History)
27 users (show)

Fixed In Version:
Doc Type: Enhancement
Doc Text:
Clone Of: 953343
Environment:
Last Closed: 2014-01-27 07:48:23 UTC
oVirt Team: ---
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
oVirt gerrit 23539 None None None Never
oVirt gerrit 23540 None None None Never
oVirt gerrit 23541 None None None Never
oVirt gerrit 23542 None None None Never

Description Sean Cohen 2014-01-15 23:10:42 UTC
+++ This bug was initially created as a clone of Bug #953343 +++

Description of problem:

Hypervisor RHEV-H only connects to the Dell Equallogic using one connection. it is expected to do multipathing and have 4 connection to it.


Version-Release number of selected component (if applicable):
rhevh-6.4-20120318.1.el6_4


How reproducible:


Steps to Reproduce:
1. install rhev hypervisor
2. connect to rhev-m
3. configure iscsi storage
  
Actual results:

hypervisor only have one connection to Equallogic (i.e not using multipath)

Expected results:

suppose to have 4 connection using multipathing to the Dell Equallogic

Additional info:

=======================
the way that this was "fixed" (probably not the optimal config, but it works)

configure from console (F2) of the Hypervisor

-----------------------------------------------------------------------
[root@vh-7 network-scripts]# vi /var/lib/iscsi/ifaces/iSCSI:1
# BEGIN RECORD 6.2.0-873.2.el6
iface.iscsi_ifacename = iSCSI:1
iface.net_ifacename = iSCSI
iface.transport_name = tcp
iface.vlan_id = 0
iface.vlan_priority = 0
iface.iface_num = 0
iface.mtu = 0
iface.port = 0
# END RECORD

-----------------------------------------------------------------------
[root@vh-7 network-scripts]# vi /var/lib/iscsi/ifaces/iSCSI:2
# BEGIN RECORD 6.2.0-873.2.el6
iface.iscsi_ifacename = iSCSI:2
iface.net_ifacename = iSCSI
iface.transport_name = tcp
iface.vlan_id = 0
iface.vlan_priority = 0
iface.iface_num = 0
iface.mtu = 0
iface.port = 0
# END RECORD

-----------------------------------------------------------------------
[root@vh-7 admin]# vi /var/lib/iscsi/ifaces/iSCSI:3
# BEGIN RECORD 6.2.0-873.2.el6
iface.iscsi_ifacename = iSCSI:3
iface.net_ifacename = iSCSI
iface.transport_name = tcp
iface.vlan_id = 0
iface.vlan_priority = 0
iface.iface_num = 0
iface.mtu = 0
iface.port = 0
# END RECORD

-----------------------------------------------------------------------
# Add the folowing under section "devices {}"
[root@vh-7 ~]# vi /etc/multipath.conf
device {
	vendor			"EQLOGIC"
	product			"100E-00"
	path_grouping_policy	multibus
	getuid_callout          "/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/%n"
	features		"1 queue_if_no_path"
	path_checker		readsector0
	path_selector		"round-robin 0"
	failback		"immediate"
	rr_min_io		10
	rr_weight		priorities
}

-----------------------------------------------------------------------
vi /etc/mount.sh
#!/bin/sh

sleep 2m

iscsiadm -m discovery -t st -p 192.168.XXX.XXX:3260

iscsiadm -m node -l -T iqn.2001-05.com.equallogic:0-XXXXX-XXXX-XXXXXXXa-stor -p 192.168.XXX.XXX:3260

-----------------------------------------------------------------------
vi /etc/rc.local
nohup sh /etc/mount.sh > /dev/null &


-----------------------------------------------------------------------
vi /config/files
/etc/rc.local
/etc/mount.sh
/etc/multipath.conf
/var/lib/iscsi/ifaces/iSCSI:1
/var/lib/iscsi/ifaces/iSCSI:2
/var/lib/iscsi/ifaces/iSCSI:3

-----------------------------------------------------------------------
chmod +x /etc/mount.sh
cp /etc/rc.local /config/etc/rc.local
cp /etc/mount.sh /config/etc/mount.sh
cp /etc/multipath.conf /config/etc/multipath.conf
cp /var/lib/iscsi/ifaces/iSCSI:1 /config/var/lib/iscsi/ifaces/iSCSI:1
cp /var/lib/iscsi/ifaces/iSCSI:2 /config/var/lib/iscsi/ifaces/iSCSI:2
cp /var/lib/iscsi/ifaces/iSCSI:3 /config/var/lib/iscsi/ifaces/iSCSI:3

--- Additional comment from Mike Burns on 2013-04-18 09:04:11 EDT ---

We do have RFE's filed (don't have the numbers handy) for allowing changes to multipath.conf from the TUI.  

How does RHEL/Fedora work?  Do you have to write those files manually?

This seems more like something that needs to be configured in the oVirt Engine/RHEV-M configuration and pushed down to all hypervisors, rather than in ovirt-node/RHEV-H directly.

--- Additional comment from Simon Kong Win Chang on 2013-04-18 19:13:10 EDT ---

i configured the storage on the manager first, then the hypervisor was only have one connection to the Dell EqualLogic.

i then did the modification above to have it have 4 connection to the Equallogic.

yes the config should be from the manager, hence why i submitted this. also to help anyone with an equallogic make their setup work while waiting for this feature to be implemented to the manager.

also, the disadvantage now, is, every time i am to upgrade the hypervisor, i will need to manually go in and change those config again.

we are running the "Red Hat Enterprise Virtualization Manager" but i don't have access to submit a bug in that section, hence why i submitted it in the ovirt section.

--- Additional comment from Mike Burns on 2013-04-25 11:17:20 EDT ---

Moving this to ovirt-engine for triage.

As for issues with RHEV-M, you should file a support ticket with RH Support.  They can get a bug filed for you.

--- Additional comment from Simon Kong Win Chang on 2013-04-25 19:26:05 EDT ---

i have, case number 00811800

--- Additional comment from Simon Kong Win Chang on 2013-05-08 20:07:22 EDT ---

p.s upgrading the hypervisor using the RHEV-Manager did not wipe the config. still worked fine after the upgrade.

--- Additional comment from Simon Kong Win Chang on 2013-05-08 20:12:17 EDT ---

added the 2 mkdir line in this section


chmod +x /etc/mount.sh
cp /etc/rc.local /config/etc/rc.local
cp /etc/mount.sh /config/etc/mount.sh
cp /etc/multipath.conf /config/etc/multipath.conf
==mkdir /config/var/lib/iscsi==
==mkdir /config/var/lib/iscsi/ifaces==
cp /var/lib/iscsi/ifaces/iSCSI:1 /config/var/lib/iscsi/ifaces/iSCSI:1
cp /var/lib/iscsi/ifaces/iSCSI:2 /config/var/lib/iscsi/ifaces/iSCSI:2
cp /var/lib/iscsi/ifaces/iSCSI:3 /config/var/lib/iscsi/ifaces/iSCSI:3

--- Additional comment from Sebastian Antunez on 2013-06-23 13:53:14 EDT ---

Hello

I have installed RHEV 3.1 connected to an Equallogic storage, but only a connection is active. When configuring RHEV as the standard process under Redhat problem persists.

Is there any additional procedure to have 2 visible and not one path.

Regards

--- Additional comment from Simon Kong Win Chang on 2013-06-24 21:39:41 EDT ---

the workaround posted above is to add 3 path, therefore a total of 4.

if you only need two path, create all the above except for /var/lib/iscsi/ifaces/iSCSI:2 and /var/lib/iscsi/ifaces/iSCSI:3

--- Additional comment from Donald Williams on 2013-07-01 22:31:00 EDT ---

One thing I didn't see is the contents of the /etc/sysctl.conf file. 

When you have multiple NICs on the same subnet you need to add the following to the /etc/sysctl.conf file. 

As a test you should be able to ping -I <iscsi eth ports> <EQL Group IP>  Try all the ports. Without the settings below, only one interface will work. 

# EQL entries
net.ipv4.conf.all.arp_ignore = 1
net.ipv4.conf.all.arp_announce = 2
net.ipv4.conf.all.rp_filter = 2

Optionally I have found the following helpful with Oracle OVMS and SQL servers. 
Also added to the /etc/sysctl.conf

# Increase network buffer sizes
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.ipv4.tcp_rmem = 8192 87380 16777216
net.ipv4.tcp_wmem = 4096 65536 16777216
net.core.wmem_default = 262144
net.core.rmem_default = 262144

Run sysctl -p afterwards and try the ping again.  If that works then re-do the Discovery and login.

--- Additional comment from Donald Williams on 2013-07-01 22:37:41 EDT ---

Something else I just noticed, you ifaces iscsi_ifacenanme & net_ifacename aren't the same. I've never seen that work. Though I'm used to RHEL or Oracle Linux, ubuntu, etc... 

iface.iscsi_ifacename = iSCSI:3
iface.net_ifacename = iSCSI

On a standard config they are: 

# cat /etc/iscsi/ifaces/eth0
# BEGIN RECORD 2.0-873
iface.iscsi_ifacename = eth0
iface.net_ifacename = eth0
iface.transport_name = tcp
iface.vlan_id = 0
iface.vlan_priority = 0
iface.iface_num = 0
iface.mtu = 0
iface.port = 0
# END RECORD

--- Additional comment from Simon Kong Win Chang on 2013-07-02 01:31:04 EDT ---

Sorry, i should have given a bit more network configuration information with that.


the "iSCSI" is actually the name of the network interface which is configured with an ip address. the iSCSI interface is actually a bridge interface build on top of a LACP bond interface, which in turn is build on top of the physical interface. the config might make is less confusing.

emX interface are the physical interface.

bondX.X are the LACP bonding interface. we are actually bonding 4 physical interface.

iSCSI interface is actually the bridge interface. it could have been labled as br0.130 instead. if this was the case, then the config files would have been

# cat /etc/iscsi/ifaces/br0.130:1
# cat /etc/iscsi/ifaces/br0.130:2
# cat /etc/iscsi/ifaces/br0.130:3

the ":X" after the interface are usually use to configure alias interface in the network config. although in this case, no alias interface was configured in /etc/sysconfig/network-scripts/


=============network config files==============

# cat /etc/sysconfig/network-scripts/ifcfg-em1
DEVICE=em1
ONBOOT=yes
HWADDR=XX:XX:XX:XX:XX:XX
MASTER=bond0
SLAVE=yes
MTU=9000
NM_CONTROLLED=no
STP=no
PEERDNS=no

# cat /etc/sysconfig/network-scripts/ifcfg-em2
DEVICE=em2
ONBOOT=yes
HWADDR=XX:XX:XX:XX:XX:XX
MASTER=bond0
SLAVE=yes
MTU=9000
NM_CONTROLLED=no
STP=no
PEERDNS=no

# cat /etc/sysconfig/network-scripts/ifcfg-bond0
DEVICE=bond0
ONBOOT=yes
BONDING_OPTS=mode=4
MTU=9000
NM_CONTROLLED=no
STP=no
PEERDNS=no

# cat /etc/sysconfig/network-scripts/ifcfg-bond0.130
DEVICE=bond0.130
ONBOOT=yes
VLAN=yes
BRIDGE=iSCSI
MTU=9000
NM_CONTROLLED=no
STP=no
PEERDNS=no

# cat /etc/sysconfig/network-scripts/ifcfg-iSCSI
DEVICE=iSCSI
ONBOOT=yes
TYPE=Bridge
DELAY=0
IPADDR=192.168.XXX.XXX
NETMASK=255.255.255.0
BOOTPROTO=none
MTU=9000
NM_CONTROLLED=no
STP=no
PEERDNS=no

====================================================
p.s ping works fine

a small extract of the output of
# iscsiadm -m session -P3

Target: iqn.2001-05.com.equallogic:0-XXXX
        Current Portal: 192.168.XXX.201:3260,1
        Persistent Portal: 192.168.XXX.200:3260,1
                **********
                Interface:
                **********
                Iface Name: default
                Iface Transport: tcp
                Iface Initiatorname: iqn.1994-05.com.XXXX
                Iface IPaddress: 192.168.XXX.148
                Iface HWaddress: <empty>
                Iface Netdev: <empty>
                SID: 1
                iSCSI Connection State: LOGGED IN
                iSCSI Session State: LOGGED_IN
.
.
.
       Current Portal: 192.168.XXX.202:3260,1
        Persistent Portal: 192.168.XXX.200:3260,1
                **********
                Interface:
                **********
                Iface Name: iSCSI:1
                Iface Transport: tcp
                Iface Initiatorname: iqn.1994-05.com.XXXX
                Iface IPaddress: 192.168.XXX.148 
                Iface HWaddress: <empty> 
                Iface Netdev: iSCSI
                SID: 2
.
.
.
        Current Portal: 192.168.XXX.203:3260,1
        Persistent Portal: 192.168.XXX.200:3260,1
                **********
                Interface:
                **********
                Iface Name: iSCSI:2
                Iface Transport: tcp
                Iface Initiatorname: iqn.1994-05.com.XXXX
                Iface IPaddress: 192.168.XXX.148
                Iface HWaddress: <empty>
                Iface Netdev: iSCSI
                SID: 3
.
.
.
        Current Portal: 192.168.XXX.204:3260,1
        Persistent Portal: 192.168.XXX.200:3260,1
                **********
                Interface:
                **********
                Iface Name: iSCSI:3
                Iface Transport: tcp
                Iface Initiatorname: iqn.1994-05.com.XXXX
                Iface IPaddress: 192.168.XXX.148
                Iface HWaddress: <empty>
                Iface Netdev: iSCSI
                SID: 4
.
.
.


===============
statistics info from the equallogic confirms that there is data transfer between all the interface.

hope this helps.

--- Additional comment from Simon Kong Win Chang on 2013-07-02 01:33:19 EDT ---

just to make is less confusing
the 192.168.XXX.148 is assagined to the iSCSI network interface


# cat /etc/sysconfig/network-scripts/ifcfg-iSCSI
DEVICE=iSCSI
ONBOOT=yes
TYPE=Bridge
DELAY=0
IPADDR=192.168.XXX.148
NETMASK=255.255.255.0
BOOTPROTO=none
MTU=9000
NM_CONTROLLED=no
STP=no
PEERDNS=no

--- Additional comment from Donald Williams on 2013-07-02 09:30:27 EDT ---

I have never heard of anyone tring to run MPIO over a bonded interface with an EQL array.  That does explain why ping works.

You're not generating new iSCSI sessions. EQL doesn't support bonding, so traffic over that bond isn't being spread over the available ports. One session ends up on one physical interface on the group. 

With access to each interface, open-iscsi will be able create unique sessions over those ports. Then devmapper can create multipath devices correct. 

I have two NICs definted for iSCSI.  eth0/eth1
When I discover them I get one entry for each defined interface. 

for example: 
# iscsiadm -m node
172.23.151.120:3260,1 iqn.2001-05.com.equallogic:4-52aed6-d8e3e0689-74c004a89f851c27-ubuntu-4k-vol-test
172.23.151.120:3260,1 iqn.2001-05.com.equallogic:4-52aed6-d8e3e0689-74c004a89f851c27-ubuntu-4k-vol-test

When I login, I get two logins, so now I have two iSCSI connections. 

iscsiadm -m node -T iqn.2001-05.com.equallogic:4-52aed6-d8e3e0689-74c004a89f851c27-ubuntu-4k-vol-test -l
Logging in to [iface: eth0, target: iqn.2001-05.com.equallogic:4-52aed6-d8e3e0689-74c004a89f851c27-ubuntu-4k-vol-test, portal: 172.23.151.120,3260] (multiple)
Logging in to [iface: eth1, target: iqn.2001-05.com.equallogic:4-52aed6-d8e3e0689-74c004a89f851c27-ubuntu-4k-vol-test, portal: 172.23.151.120,3260] (multiple)
Login to [iface: eth0, target: iqn.2001-05.com.equallogic:4-52aed6-d8e3e0689-74c004a89f851c27-ubuntu-4k-vol-test, portal: 172.23.151.120,3260] successful.
Login to [iface: eth1, target: iqn.2001-05.com.equallogic:4-52aed6-d8e3e0689-74c004a89f851c27-ubuntu-4k-vol-test, portal: 172.23.151.120,3260] successful.


#multipath -ll
mpath4 (364ed2a9568e0e3d8271c859fa804c074) dm-2 EQLOGIC,100E-00
size=20G features='0' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
  |- 15:0:0:0 sdd 8:48 active ready  running
  `- 16:0:0:0 sde 8:64 active ready  running

Now multipathd will round robin IO across those two iSCSI sessions. 

Compare my session output to yours. 

# iscsiadm -m session -P3
iSCSI Transport Class version 2.0-870
version 2.0-873
Target: iqn.2001-05.com.equallogic:4-52aed6-d8e3e0689-74c004a89f851c27-ubuntu-4k-vol-test
        Current Portal: 172.23.151.121:3260,1
        Persistent Portal: 172.23.151.120:3260,1
                **********
                Interface:
                **********
                Iface Name: eth0
                Iface Transport: tcp
                Iface Initiatorname: iqn.1993-08.org.debian:8ab9cf5340f0
                Iface IPaddress: 172.23.71.231
                Iface HWaddress: <empty>
                Iface Netdev: eth0
                SID: 11
                iSCSI Connection State: LOGGED IN
                iSCSI Session State: LOGGED_IN
                Internal iscsid Session State: NO CHANGE
                *********
                Timeouts:
                *********
                Recovery Timeout: 120
                Target Reset Timeout: 30
                LUN Reset Timeout: 30
                Abort Timeout: 15
                *****
                CHAP:
                *****
                username: <empty>
                password: ********
                username_in: <empty>
                password_in: ********
                ************************
                Negotiated iSCSI params:
                ************************
                HeaderDigest: None
                DataDigest: None
                MaxRecvDataSegmentLength: 262144
                MaxXmitDataSegmentLength: 65536
                FirstBurstLength: 65536
                MaxBurstLength: 262144
                ImmediateData: Yes
                InitialR2T: No
                MaxOutstandingR2T: 1
                ************************
                Attached SCSI devices:
                ************************
                Host Number: 15 State: running
                scsi15 Channel 00 Id 0 Lun: 0
                        Attached scsi disk sdd          State: running
        Current Portal: 172.23.151.122:3260,1
        Persistent Portal: 172.23.151.120:3260,1
                **********
                Interface:
                **********
                Iface Name: eth1
                Iface Transport: tcp
                Iface Initiatorname: iqn.1993-08.org.debian:8ab9cf5340f0
                Iface IPaddress: 172.23.74.186
                Iface HWaddress: <empty>
                Iface Netdev: eth1
                SID: 12
                iSCSI Connection State: LOGGED IN
                iSCSI Session State: LOGGED_IN
                Internal iscsid Session State: NO CHANGE
                *********
                Timeouts:
                *********
                Recovery Timeout: 120
                Target Reset Timeout: 30
                LUN Reset Timeout: 30
                Abort Timeout: 15
                *****
                CHAP:
                *****
                username: <empty>
                password: ********
                username_in: <empty>
                password_in: ********
                ************************
                Negotiated iSCSI params:
                ************************
                HeaderDigest: None
                DataDigest: None
                MaxRecvDataSegmentLength: 262144
                MaxXmitDataSegmentLength: 65536
                FirstBurstLength: 65536
                MaxBurstLength: 262144
                ImmediateData: Yes
                InitialR2T: No
                MaxOutstandingR2T: 1
                ************************
                Attached SCSI devices:
                ************************
                Host Number: 16 State: running
                scsi16 Channel 00 Id 0 Lun: 0
                        Attached scsi disk sde          State: running

--- Additional comment from Donald Williams on 2013-07-02 13:33:40 EDT ---

OK, I see what you did, but why do you have to create the BOND?  Why can't you use the network interfaces via their HyperVisor object names?

Comment 1 Sergey Gotliv 2014-01-25 18:52:33 UTC
*** Bug 953343 has been marked as a duplicate of this bug. ***

Comment 2 Itamar Heim 2014-01-27 07:48:23 UTC

*** This bug has been marked as a duplicate of bug 753541 ***


Note You need to log in before you can comment on or make changes to this bug.