Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 1363864 - Libvirt does not change the ownership of image files automatically to the qemu user /group
Summary: Libvirt does not change the ownership of image files automatically to the qem...
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: Virtualization Tools
Classification: Community
Component: libvirt
Version: unspecified
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Michal Privoznik
QA Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-08-03 18:25 UTC by Jonatan Schlag
Modified: 2016-09-21 11:53 UTC (History)
8 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2016-09-21 11:53:54 UTC


Attachments (Terms of Use)

Description Jonatan Schlag 2016-08-03 18:25:36 UTC
Description of problem:

On a lot of systems (Debian, Arch Linux), I use libvirt change the ownership of image files dynamically to the qemu user and the qemu group. Now I compiled and packaged libvirt for a system and the ownership change did not work. The filesystem (ext4) is writeable but libvirt does not change the ownership.

Version-Release number of selected component (if applicable):
1.3.3.2

Steps to Reproduce:
1. Create a new image file (ownership is root:root)
2. Attach the image file to a VM
3. Try to start the VM

Actual results:
The VM fails to start with the following error:

libvirtError: internal error: process exited while connecting to monitor: able-ticketing,seamless-migration=on -device qxl-vga,id=video0,ram_size=67108864,vram_size=67108864,vram64_size_mb=0,vgamem_mb=16,bus=pci.0,addr=0x2 -device intel-hda,id=sound0,bus=pci.0,addr=0x4 -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 -chardev spicevmc,id=charredir0,name=usbredir -device usb-redir,chardev=charredir0,id=redir0 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x7 -msg timestamp=on
2016-08-03T18:19:47.494512Z qemu-system-x86_64: -drive file=/data/hdd1/libvirt/images/test.img,format=raw,if=none,id=drive-virtio-disk0: Could not open '/data/hdd1/libvirt/images/test.img': Permission denied

Expected results:
LIbvirt should change the ownership of the file /data/hdd1/libvirt/images/test.img to nobody:kvm (qemu user and qemu group) and the VM should start


Additional info:
gcc: 4.9.4
glibc: 2.12

Comment 1 Cole Robinson 2016-08-03 20:12:37 UTC
Thanks for the report. You probably are missing the --with-qemu-user/group libvirt configure options. This is what distribution packages use to tell libvirt to run qemu binaries as qemu:qemu, and change file ownership to match. Otherwise the default is root:root, which is kind of problematic because libvirt will drop privs for the emulator which causes issues.

$ ./configure --help | grep qemu
  --with-qemu             add QEMU/KVM support [default=yes]
  --with-qemu-user        username to run QEMU system instance as
  --with-qemu-group       groupname to run QEMU system instance as


So, pass: --with-qemu-user qemu --with-qemu-group qemu

Please reopen if I'm wrong

Comment 2 Michael Tremer 2016-08-03 21:54:45 UTC
Thanks Cole for looking into this.

Actually libvirt is compiled with --with-qemu-user=nobody --with-qemu-group=kvm. So that should essentially do the same as qemu:qemu, don't you think?

Comment 3 Laine Stump 2016-08-03 22:03:18 UTC
Set it in /etc/libvirt/qemu.conf - that will override whatever is compiled in.

Comment 4 Cole Robinson 2016-08-03 22:34:51 UTC
Sorry I didn't read closely enough. qemu:qemu is what fedora uses. So if you passed --with-qemu-user=nobody --with-qemu-group=kvm to ./configure, and that is what matches Arch linux package config, I think it should be working. Do you have any custom settings in /etc/libvirt/qemu.conf that might be changing behavior unexpectedly?

Comment 5 Jonatan Schlag 2016-08-04 07:27:41 UTC
Hi, 
thanks for any help. 

Libvirt is configured with:

./configure --prefix=/usr --localstatedir=/var --sysconfdir=/etc \
			--with-openssl --without-sasl \
			--without-uml --without-vbox --without-lxc \
                        --without-esx  --without-vmware --without-openvz \
			--without-firewalld --without-network \
                        --with-interface --with-virtualport --with-macvtap \
			--disable-nls --without-avahi --without-test-suit \
                        --without-dbus \
			--with-qemu-user=nobody --with-qemu-group=kvm \
			--with-storage-dir --without-storage-fs \
                        --without-storage-lvm  --without-storage-iscsi \
			--without-storage-scsi --without-storage-mpath \
                        --without-storage-disk --without-storage-rbd \
                        --without-storage-sheepdog --without-storage-gluster \
                        --without-storage-zfs

The I posted the qemu.conf there:

http://nopaste.ipfire.org/view/4Aesq26x


Qemu runs with the right of nobody:kvm when it is started by libvirt. (htop says also that qemu runs as user nobody http://nopaste.ipfire.org/view/vCp1urKa) So the option works, but a qemu process with the rights of nobody:kvm can not access an image file which is owned by root:root.

On Arch Linux the ownership of the file is changed by libvirt to noboy:kvm on the system (IPFire) for it I compiled and packaged libvirt, libvirt seems to be unable to change the ownership of the image file and qemu failed with the error in my first post.

Regards Jonatan

Comment 6 Jonatan Schlag 2016-08-04 10:15:50 UTC
Hi,
i uploaded the debug log to 

http://people.ipfire.org/~jschlag/1363864/1_libvirtd.log

The UID of the user nobody is 99, the GID of the group kvm is 1011.

Following the log the ownership is changed but why is the file still owned by root:root?

The daemon run as root and there are also no selinux which prevent the access to the file.

Regards Jonatan

Comment 7 Jonatan Schlag 2016-08-04 12:03:36 UTC
Apparmor is also disabled or better not installed.

Comment 8 Michael Tremer 2016-08-04 17:40:10 UTC
We have narrowed some things down here. We found out that virStorageFileSupportsSecurityDriver() returns false for our storage driver (which is storage dir) although we expect it to return true.

virStorageFileSupportsSecurityDriver goes to the storage backend and checks if storageFileChown is set. That is the case for all storage dir and storage fs backend types.

Could anyone confirm that this behaviour is correct here or if our assumption that the wrong value is returned here is correct.

> static int
> qemuSecurityChownCallback(virStorageSourcePtr src,
>                           uid_t uid,
>                           gid_t gid)
> {
>    struct stat sb;
>    int save_errno = 0;
>    int ret = -1;
>
>    if (!virStorageFileSupportsSecurityDriver(src))
>        return 0;

Comment 9 Michal Privoznik 2016-08-05 08:13:42 UTC
(In reply to Michael Tremer from comment #8)
> We have narrowed some things down here. We found out that
> virStorageFileSupportsSecurityDriver() returns false for our storage driver
> (which is storage dir) although we expect it to return true.

Our storage driver? Are you writing your own storage driver?

> 
> virStorageFileSupportsSecurityDriver goes to the storage backend and checks
> if storageFileChown is set. That is the case for all storage dir and storage
> fs backend types.
> 
> Could anyone confirm that this behaviour is correct here or if our
> assumption that the wrong value is returned here is correct.
> 
> > static int
> > qemuSecurityChownCallback(virStorageSourcePtr src,
> >                           uid_t uid,
> >                           gid_t gid)
> > {
> >    struct stat sb;
> >    int save_errno = 0;
> >    int ret = -1;
> >
> >    if (!virStorageFileSupportsSecurityDriver(src))
> >        return 0;


I can't confirm what the return value should be unless I know what src is. From the upstream discussion I've learned that you're using storage pool and in the domain XML there's storage volume from that pool. However, depending on what the /data/path/to/your/images is (e.g. is it a NFS mount?), what the storage pool XML and domain XML look like this might return true and false.

BTW: can you attach the debugger once again, try to step into virStorageFileSupportsSecurityDriver() and look what backend points to? I'd reproduce myself, but I don't have the configuration you have.

Comment 10 Jonatan Schlag 2016-08-05 10:05:31 UTC
Hi,
thank you very much, Michal Privoznik.

I posted our domain xml file here:

http://nopaste.ipfire.org/view/QrUaJOks

the storage xml file is available here:

http://nopaste.ipfire.org/view/QVYDDjGV.

The storage pool is directly  located on a 1 TN hard drive we do not use NFS samba or other network protocols.

Comment 11 Michael Tremer 2016-08-05 15:43:50 UTC
GDB output: http://nopaste.ipfire.org/view/6MhSKdPB

We found out that backend is actually not set and hence virStorageFileSupportsSecurityDriver is returning false.

> 2716 bool
> 2717 virStorageFileSupportsSecurityDriver(virStorageSourcePtr src)
> 2718 {
> 2719     int actualType;
> 2720     virStorageFileBackendPtr backend;
> 2721 
> 2722     if (!src)
> 2723         return false;
> 2724     actualType = virStorageSourceGetActualType(src);
> 2725 
> 2726     if (src->drv) {
> 2727         backend = src->drv->backend;
> 2728     } else {
> 2729         if (!(backend = virStorageFileBackendForTypeInternal(actualType,
> 2730                                                              src->protocol,
> 2731                                                              false)))
> 2732             return false;
> 2733     }
> 2734 
> 2735     return !!backend->storageFileChown;
> 2736 }

libvirtd jumps directly into line 2729 where actualType is set to 1 (which we assume is VIR_STORAGE_TYPE_FILE). According to the configuration this should actually be VIR_STORAGE_TYPE_DIR, we *think*.

Right here we are stuck again. Please let us know if this information helps you to get closer to the problem and if we could provide any further information if needed.

Comment 12 Michal Privoznik 2016-08-09 14:34:36 UTC
(In reply to Michael Tremer from comment #11)
> GDB output: http://nopaste.ipfire.org/view/6MhSKdPB
> 
> We found out that backend is actually not set and hence
> virStorageFileSupportsSecurityDriver is returning false.
> 
> > 2716 bool
> > 2717 virStorageFileSupportsSecurityDriver(virStorageSourcePtr src)
> > 2718 {
> > 2719     int actualType;
> > 2720     virStorageFileBackendPtr backend;
> > 2721 
> > 2722     if (!src)
> > 2723         return false;
> > 2724     actualType = virStorageSourceGetActualType(src);
> > 2725 
> > 2726     if (src->drv) {
> > 2727         backend = src->drv->backend;
> > 2728     } else {
> > 2729         if (!(backend = virStorageFileBackendForTypeInternal(actualType,
> > 2730                                                              src->protocol,
> > 2731                                                              false)))
> > 2732             return false;
> > 2733     }
> > 2734 
> > 2735     return !!backend->storageFileChown;
> > 2736 }
> 
> libvirtd jumps directly into line 2729 where actualType is set to 1 (which
> we assume is VIR_STORAGE_TYPE_FILE). According to the configuration this
> should actually be VIR_STORAGE_TYPE_DIR, we *think*.
> 
> Right here we are stuck again. Please let us know if this information helps
> you to get closer to the problem and if we could provide any further
> information if needed.

I believe this is a packaging issue. I mean, you say you are building the libvirt on your own and it does not work. Moreover, in comment 5 you give us the configure command line. Well, if you enable storage fs there, everything will work again:

./configure --with-storage-fs ...

But I'll rather wait for confirmation before closing this one.

The thing is, in the configure phase you are deliberately disabling VIR_STORAGE_TYPE_FILE backend to storage driver. Therefore, virStorageFileSupportsSecurityDriver() is not fiding any backend for the type of source configured in domain XML and thus do not relabel it.

Comment 13 Jonatan Schlag 2016-08-22 15:40:14 UTC
Hi,
Sorry for the delay, but my last 2 weeks were full with other work.

If I enable storage fs everything works fine, thank you very much.

This solves my issue, but I think there is still a bug because libvirt do not log that it fails to change the ownership.

There should be something like

info: Unable to change the ownership because libvirt was compiled without storage fs.

In the momenthe log is:

2016-08-04 09:54:23.203+0000: 31063: debug : qemuProcessStart:4863 : Setting domain security labels
2016-08-04 09:54:23.203+0000: 31063: info : virSecurityDACSetOwnershipInternal:248 : Setting DAC user and group on '/data/hdd1/libvirt/iso/KNOPPIX_V7.6.1DVD-2016-01-16-DE.iso' to '99:1011'
2016-08-04 09:54:23.203+0000: 31063: info : virSecurityDACSetOwnershipInternal:248 : Setting DAC user and group on '/data/hdd1/libvirt/images/test.img' to '99:1011'
2016-08-04 09:54:23.203+0000: 31063: debug : qemuProcessStart:4892 : Labelling done, completing handshake to child

But this problem should be tracked in another bug report.

So Thank you very much for every help.

Regards Jonatan

Comment 14 Michal Privoznik 2016-09-21 11:53:54 UTC
Closing per comment 13.


Note You need to log in before you can comment on or make changes to this bug.