Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 1356606 - ISO Domain not being added
Summary: ISO Domain not being added
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: Red Hat Enterprise Virtualization Manager
Classification: Red Hat
Component: ovirt-engine
Version: 4.0.2
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: ovirt-4.0.4
: ---
Assignee: Liron Aravot
QA Contact: Aharon Canan
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-07-14 12:47 UTC by Bill Sanford
Modified: 2016-07-26 15:00 UTC (History)
12 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2016-07-26 15:00:00 UTC
oVirt Team: Storage


Attachments (Terms of Use)
/var/log/ovirt-engine/engine.log (deleted)
2016-07-15 16:36 UTC, Bill Sanford
no flags Details
Host 1 of 3 (deleted)
2016-07-25 17:37 UTC, Bill Sanford
no flags Details
Host 2 of 3 (deleted)
2016-07-25 17:38 UTC, Bill Sanford
no flags Details
Host 3 of 3 (deleted)
2016-07-25 17:39 UTC, Bill Sanford
no flags Details
/var/log/messages file on host (deleted)
2016-07-26 13:07 UTC, Bill Sanford
no flags Details

Description Bill Sanford 2016-07-14 12:47:44 UTC
Description of problem: When I try and add the ISO Domain to RHEV-M after the Storage Domain, I get the error: "Failed to retrieve existing storage domain information." 

This address has an NFS mount that was previously used with a 3.6 RHEV-M install, including the directory structure that RHEV-M creates.


Version-Release number of selected component (if applicable):
RHEL - http://download.lab.bos.redhat.com/rel-eng/repos/rhel-7.2-z/x86_64/
RHEV-M - http://bob.eng.lab.tlv.redhat.com/builds/4.0/4.0.1-1/el7/

How reproducible:
100%

Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:

Comment 2 Allon Mureinik 2016-07-14 16:30:20 UTC
Can you attach the logs please?

Comment 3 Bill Sanford 2016-07-15 14:06:31 UTC
I just tailed the engine.log when I tried to import the domain:

2016-07-15 10:04:46,845 INFO  [org.ovirt.engine.core.bll.storage.connection.AddStorageServerConnectionCommand] (default task-11) [16b36da4] Lock Acquired to object 'EngineLock:{exclusiveLocks='[10.19.168.10:/rhev/isos=<STORAGE_CONNECTION, ACTION_TYPE_FAILED_OBJECT_LOCKED>]', sharedLocks='null'}'
2016-07-15 10:04:46,875 INFO  [org.ovirt.engine.core.bll.storage.connection.AddStorageServerConnectionCommand] (default task-11) [16b36da4] Running command: AddStorageServerConnectionCommand internal: false. Entities affected :  ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group CREATE_STORAGE_DOMAIN with role type ADMIN
2016-07-15 10:04:46,876 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (default task-11) [16b36da4] START, ConnectStorageServerVDSCommand(HostName = corrin, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='42f17868-584f-4c1f-8e2c-c448e1bd16d2', storagePoolId='00000000-0000-0000-0000-000000000000', storageType='NFS', connectionList='[StorageServerConnections:{id='null', connection='10.19.168.10:/rhev/isos', iqn='null', vfsType='null', mountOptions='null', nfsVersion='V3', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 7ab0c6fd

Comment 4 Bill Sanford 2016-07-15 16:36:05 UTC
Created attachment 1180206 [details]
/var/log/ovirt-engine/engine.log

Comment 5 Yaniv Kaul 2016-07-15 19:14:41 UTC
(In reply to Bill Sanford from comment #4)
> Created attachment 1180206 [details]
> /var/log/ovirt-engine/engine.log

1. Please compress logs next time.
2. Please look more closely at the engine.log, specifically:
2016-07-15 10:06:52,026 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (default task-11) [16b36da4] FINISH, ConnectStorageServerVDSCommand, return: {00000000-0000-0000-0000-000000000000=477}, log id: 7ab0c6fd
2016-07-15 10:06:52,029 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-11) [16b36da4] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: The error message for connection 10.19.168.10:/rhev/isos returned by VDSM was: Problem while trying to mount target
2016-07-15 10:06:52,030 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (default task-11) [16b36da4] The connection with details '10.19.168.10:/rhev/isos' failed because of error code '477' and error message is: problem while trying to mount target

3. And therefore, please provide VDSM logs (as we always ask when it is a storage related issue)

Comment 7 Bill Sanford 2016-07-25 17:36:51 UTC
From my laptop, I can mount the drive that RHEV-M seems incapable of doing:

[root@dhcp-10-19-63-202 ~]# sshfs -o allow_other root@10.19.168.10:/rhev/isos /home/bsanford/ns1
root@10.19.168.10's password: 
[root@dhcp-10-19-63-202 ~]# cd /home/bsanford/ns1/
[root@dhcp-10-19-63-202 ns1]# ls
cac76132-083f-4f5b-911c-827891bf657a
[root@dhcp-10-19-63-202 ns1]#

Comment 8 Bill Sanford 2016-07-25 17:37:49 UTC
Created attachment 1183906 [details]
Host 1 of 3

Comment 9 Bill Sanford 2016-07-25 17:38:21 UTC
Created attachment 1183907 [details]
Host 2 of 3

Comment 10 Bill Sanford 2016-07-25 17:39:41 UTC
Created attachment 1183908 [details]
Host 3 of 3

Comment 11 Yaniv Kaul 2016-07-25 17:59:50 UTC
(In reply to Bill Sanford from comment #7)
> From my laptop, I can mount the drive that RHEV-M seems incapable of doing:
> 
> [root@dhcp-10-19-63-202 ~]# sshfs -o allow_other

How is sshfs related here? RHEV is trying to mount with NFS.

Comment 12 Allon Mureinik 2016-07-26 08:01:10 UTC
The relevant error I can see is:

jsonrpc.Executor/5::ERROR::2016-07-25 08:37:44,228::hsm::2403::Storage.HSM::(connectStorageServer) Could not connect to storageServer
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/hsm.py", line 2400, in connectStorageServer
    conObj.connect()
  File "/usr/share/vdsm/storage/storageServer.py", line 456, in connect
    return self._mountCon.connect()
  File "/usr/share/vdsm/storage/storageServer.py", line 238, in connect
    six.reraise(t, v, tb)
  File "/usr/share/vdsm/storage/storageServer.py", line 230, in connect
    self._mount.mount(self.options, self._vfsType, cgroup=self.CGROUP)
  File "/usr/lib/python2.7/site-packages/vdsm/storage/mount.py", line 225, in mount
    timeout=timeout, cgroup=cgroup)
  File "/usr/lib/python2.7/site-packages/vdsm/supervdsm.py", line 53, in __call__
    return callMethod()
  File "/usr/lib/python2.7/site-packages/vdsm/supervdsm.py", line 51, in <lambda>
    **kwargs)
  File "<string>", line 2, in mount
  File "/usr/lib64/python2.7/multiprocessing/managers.py", line 773, in _callmethod
    raise convert_to_error(kind, result)
MountError: (32, ';mount.nfs: Connection timed out\n')


Is /rhev/isos really exposed via ssh instead of nfs?

Additionally, can your share /var/log/messages? That may shed some light on the connection timing out.

Comment 13 Bill Sanford 2016-07-26 13:07:02 UTC
I tailed the messages log during a try to connect the ISO storage and got these 4 lines on the host:

Jul 26 08:58:06 giediprime kernel: device-mapper: table: 253:21: multipath: error getting device
Jul 26 08:58:06 giediprime kernel: device-mapper: ioctl: error adding target to table
Jul 26 08:58:06 giediprime multipathd: dm-21: remove map (uevent)
Jul 26 08:58:06 giediprime multipathd: dm-21: remove map (uevent)

Comment 14 Bill Sanford 2016-07-26 13:07:48 UTC
Created attachment 1184271 [details]
/var/log/messages file on host

Comment 15 Yaniv Kaul 2016-07-26 13:08:21 UTC
(In reply to Bill Sanford from comment #13)
> I tailed the messages log during a try to connect the ISO storage and got
> these 4 lines on the host:
> 
> Jul 26 08:58:06 giediprime kernel: device-mapper: table: 253:21: multipath:
> error getting device
> Jul 26 08:58:06 giediprime kernel: device-mapper: ioctl: error adding target
> to table
> Jul 26 08:58:06 giediprime multipathd: dm-21: remove map (uevent)
> Jul 26 08:58:06 giediprime multipathd: dm-21: remove map (uevent)

All those lines have nothing to do with NFS...

Can you sync with RHEV QE guys so they'll look at the issue?

Comment 16 Yaniv Kaul 2016-07-26 13:11:09 UTC
(In reply to Bill Sanford from comment #14)
> Created attachment 1184271 [details]
> /var/log/messages file on host

Nothing interesting here unfortunately.


Note You need to log in before you can comment on or make changes to this bug.