Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 1665553 - Unable to migrate VM's using ceph storage - Unsafe migration: Migration without shared storage is unsafe [NEEDINFO]
Summary: Unable to migrate VM's using ceph storage - Unsafe migration: Migration witho...
Keywords:
Status: ON_QA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: libvirt
Version: 7.6
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: rc
: 7.6
Assignee: Michal Privoznik
QA Contact: gaojianan
URL:
Whiteboard:
: 1667037 (view as bug list)
Depends On:
Blocks: 1672178
TreeView+ depends on / blocked
 
Reported: 2019-01-11 18:28 UTC by amashah
Modified: 2019-04-12 12:24 UTC (History)
15 users (show)

Fixed In Version: libvirt-4.5.0-11.el7
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1672178 (view as bug list)
Environment:
Last Closed:
Target Upstream Version:
mprivozn: needinfo? (amashah)


Attachments (Terms of Use)

Description amashah 2019-01-11 18:28:47 UTC
Description of problem:

Unable to migrate VM's using ceph storage

It seems similar to this, but here ceph storage is in use.

-  https://bugzilla.redhat.com/show_bug.cgi?id=1632711


Version-Release number of selected component (if applicable):
4.2.7-5

How reproducible:

Uncertain, resources to test in a test environment are unavailable. The issue is from a customer support case.

Steps to Reproduce:
1. Create a VM
2. VM storage is on ceph storage domain
3. Try to migrate the VM to another host

Actual results:

Unable to migrate VM's


Expected results:

VM's should be able to migrate.


Additional info:

mount info:

~~~
192.168.x.x:/ovirt/data on /rhev/data-center/mnt/192.168.x.x:_ovirt_data type ceph (rw,noatime,name=cephfs,secret=<hidden>,acl,wsize=16777216)
~~~

stat info:

~~~
[root@ovirt04 ~]# stat -f /rhev/data-center/mnt/192.168.x.x:_ovirt_data
   File: "/rhev/data-center/mnt/192.168.x.x:_ovirt_data"
     ID: 16e4002558925fd0 Namelen: 255     Type: ceph
Block size: 4194304    Fundamental block size: 4194304
Blocks: Total: 32740938   Free: 31486538   Available: 31486538
Inodes: Total: 15989285   Free: -1
~~~


error from host:

~~~
2018-12-24 12:44:57,619-0600 ERROR (migsrc/d28597b1) [virt.vm] (vmId='d28597b1-d133-44fd-ab5f-1bc4ff8280e4') Unsafe migration: Migration without shared storage is unsafe (migration:290)
2018-12-24 12:44:57,864-0600 ERROR (migsrc/d28597b1) [virt.vm] (vmId='d28597b1-d133-44fd-ab5f-1bc4ff8280e4') Failed to migrate (migration:455)
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/vdsm/virt/migration.py", line 437, in _regular_run
    self._startUnderlyingMigration(time.time())
  File "/usr/lib/python2.7/site-packages/vdsm/virt/migration.py", line 509, in _startUnderlyingMigration
    self._perform_with_conv_schedule(duri, muri)
  File "/usr/lib/python2.7/site-packages/vdsm/virt/migration.py", line 587, in _perform_with_conv_schedule
    self._perform_migration(duri, muri)
  File "/usr/lib/python2.7/site-packages/vdsm/virt/migration.py", line 529, in _perform_migration
    self._migration_flags)
  File "/usr/lib/python2.7/site-packages/vdsm/virt/virdomain.py", line 98, in f
    ret = attr(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/vdsm/common/libvirtconnection.py", line 130, in wrapper
    ret = f(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/vdsm/common/function.py", line 92, in wrapper
    return func(inst, *args, **kwargs)
  File "/usr/lib64/python2.7/site-packages/libvirt.py", line 1779, in migrateToURI3
    if ret == -1: raise libvirtError ('virDomainMigrateToURI3() failed', dom=self)
libvirtError: Unsafe migration: Migration without shared storage is unsafe
~~~

Comment 3 Martin Tessun 2019-01-14 09:29:34 UTC
Hi Jarda,

this looks like ceph is not correctly identified as a shared storage:
192.168.x.x:/ovirt/data on /rhev/data-center/mnt/192.168.x.x:_ovirt_data type ceph (rw,noatime,name=cephfs,secret=<hidden>,acl,wsize=16777216)

 and the error in libvirt:
libvirtError: Unsafe migration: Migration without shared storage is unsafe

As you can see the ceph storage is shared.

Thanks for looking into this.

Comment 4 Han Han 2019-01-16 03:14:06 UTC
(In reply to Martin Tessun from comment #3)
> Hi Jarda,
> 
> this looks like ceph is not correctly identified as a shared storage:
> 192.168.x.x:/ovirt/data on /rhev/data-center/mnt/192.168.x.x:_ovirt_data
> type ceph (rw,noatime,name=cephfs,secret=<hidden>,acl,wsize=16777216)
> 
>  and the error in libvirt:
> libvirtError: Unsafe migration: Migration without shared storage is unsafe
> 
> As you can see the ceph storage is shared.
> 
> Thanks for looking into this.

Currently libvirt doesn't identify cephfs as shared fs:
  3764 int virFileIsSharedFS(const char *path)                                         
  3765 {                                                                               
  3766     return virFileIsSharedFSType(path,                                          
  3767     ┆   ┆   ┆   ┆   ┆   ┆   ┆   ┆VIR_FILE_SHFS_NFS |                            
  3768     ┆   ┆   ┆   ┆   ┆   ┆   ┆   ┆VIR_FILE_SHFS_GFS2 |                           
  3769     ┆   ┆   ┆   ┆   ┆   ┆   ┆   ┆VIR_FILE_SHFS_OCFS |                           
  3770     ┆   ┆   ┆   ┆   ┆   ┆   ┆   ┆VIR_FILE_SHFS_AFS |                            
  3771     ┆   ┆   ┆   ┆   ┆   ┆   ┆   ┆VIR_FILE_SHFS_SMB |                            
  3772     ┆   ┆   ┆   ┆   ┆   ┆   ┆   ┆VIR_FILE_SHFS_CIFS);                           
  3773 }   

The current shared FSes are nfs,gfs2(including glusterfs.fuse,ocfs,afs,smb,cifs)

Comment 5 Han Han 2019-01-17 02:28:14 UTC
Hi Michal,
I think it time to add cephfs migration support in libvirt since rhv has already supported this storage.

Comment 6 Jiri Denemark 2019-01-17 11:28:01 UTC
*** Bug 1667037 has been marked as a duplicate of this bug. ***

Comment 8 Michal Privoznik 2019-01-24 08:51:25 UTC
Hi,

can you please attach /proc/mounts from the host that has the ceph mounted? Thanks.

Comment 9 Guido Langenbach 2019-01-24 15:55:54 UTC
We have the same problem. This is the output from one of our hosts:

[root@server ~]# cat /proc/mounts 
rootfs / rootfs rw 0 0
sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0
proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0
devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=98508412k,nr_inodes=24627103,mode=755 0 0
securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0
tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0
devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0
tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0
tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0
cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0
pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0
efivarfs /sys/firmware/efi/efivars efivarfs rw,nosuid,nodev,noexec,relatime 0 0
cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0
cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuacct,cpu 0 0
cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0
cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0
cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0
cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0
cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0
cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_prio,net_cls 0 0
cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0
cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0
configfs /sys/kernel/config configfs rw,relatime 0 0
/dev/mapper/rhel-root / xfs rw,seclabel,relatime,attr2,inode64,logbsize=256k,sunit=512,swidth=512,noquota 0 0
selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0
systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=34,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=32450 0 0
debugfs /sys/kernel/debug debugfs rw,relatime 0 0
mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0
hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime 0 0
hugetlbfs /dev/hugepages1G hugetlbfs rw,seclabel,relatime,pagesize=1G 0 0
/dev/mapper/gluster_vg_sda4-gluster_lv_engine /gluster_bricks/engine xfs rw,seclabel,noatime,nodiratime,attr2,inode64,logbsize=256k,sunit=512,swidth=6144,noquota 0 0
/dev/sda2 /boot xfs rw,seclabel,relatime,attr2,inode64,logbsize=256k,sunit=512,swidth=512,noquota 0 0
/dev/sda1 /boot/efi vfat rw,relatime,fmask=0077,dmask=0077,codepage=437,iocharset=ascii,shortname=winnt,errors=remount-ro 0 0
/dev/mapper/gluster_vg_sda4-gluster_lv_isos /gluster_bricks/isos xfs rw,seclabel,noatime,nodiratime,attr2,inode64,logbsize=256k,sunit=512,swidth=6144,noquota 0 0
/dev/mapper/gluster_vg_sda4-gluster_lv_vmstore /gluster_bricks/vmstore xfs rw,seclabel,noatime,nodiratime,attr2,inode64,logbsize=256k,sunit=512,swidth=6144,noquota 0 0
sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0
192.168.1.10:/engine /rhev/data-center/mnt/glusterSD/192.168.1.10:_engine fuse.glusterfs rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072 0 0
fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0
192.168.1.10:/vmstore /rhev/data-center/mnt/glusterSD/192.168.1.10:_vmstore fuse.glusterfs rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072 0 0
192.168.1.10:/isos /rhev/data-center/mnt/glusterSD/192.168.1.10:_isos fuse.glusterfs rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072 0 0
binfmt_misc /proc/sys/fs/binfmt_misc binfmt_misc rw,relatime 0 0
192.168.1.11:6789:/ /rhev/data-center/mnt/192.168.1.11:6789:_ ceph rw,relatime,name=admin,secret=<hidden>,acl,wsize=16777216 0 0
tmpfs /run/user/1174800007 tmpfs rw,seclabel,nosuid,nodev,relatime,size=19704212k,mode=700,uid=1174800007,gid=1174800007 0 0

Comment 11 Michal Privoznik 2019-01-25 15:36:09 UTC
Patch proposed upstream:

https://www.redhat.com/archives/libvir-list/2019-January/msg01008.html

Comment 12 Michal Privoznik 2019-01-28 14:08:03 UTC
Patch pushed upstream:

commit 6dd2a2ae6386b1d51edcc9a434f56d7f9dc2cb35
Author:     Michal Privoznik <mprivozn@redhat.com>
AuthorDate: Thu Jan 24 09:52:42 2019 +0100
Commit:     Michal Privoznik <mprivozn@redhat.com>
CommitDate: Mon Jan 28 14:56:21 2019 +0100

    virfile: Detect ceph as shared FS
    
    https://bugzilla.redhat.com/show_bug.cgi?id=1665553
    
    Ceph can be mounted just like any other filesystem and in fact is
    a shared and cluster filesystem. The filesystem magic constant
    was taken from kernel sources as it is not in magic.h yet.
    
    Signed-off-by: Michal Privoznik <mprivozn@redhat.com>
    Reviewed-by: Erik Skultety <eskultet@redhat.com>

v5.0.0-130-g6dd2a2ae63


Note You need to log in before you can comment on or make changes to this bug.