Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 1148712 - avc denials accessing sanlock socket while deploying hosted engine
Summary: avc denials accessing sanlock socket while deploying hosted engine
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Red Hat Enterprise Virtualization Manager
Classification: Red Hat
Component: vdsm
Version: 3.5.0
Hardware: x86_64
OS: Linux
unspecified
urgent
Target Milestone: ---
: 3.5.0
Assignee: nobody nobody
QA Contact: Nikolai Sednev
URL:
Whiteboard: virt
Depends On:
Blocks: 1067162 1149634
TreeView+ depends on / blocked
 
Reported: 2014-10-02 08:02 UTC by Nikolai Sednev
Modified: 2014-12-15 14:46 UTC (History)
14 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2014-11-05 07:36:19 UTC
oVirt Team: ---
Target Upstream Version:
stirabos: needinfo-


Attachments (Terms of Use)
logs (deleted)
2014-10-13 12:26 UTC, Nikolai Sednev
no flags Details


Links
System ID Priority Status Summary Last Updated
Red Hat Bugzilla 1146529 None None None Never

Internal Links: 1146529

Description Nikolai Sednev 2014-10-02 08:02:05 UTC
Description of problem:
Failed to deploy HE on RHEL6.6:

[root@alma03 ~]# hosted-engine --deploy                                                                        
[ INFO  ] Stage: Initializing                                                                                  
          Continuing will configure this host for serving as hypervisor and create a VM where you have to install oVirt Engine afterwards.
          Are you sure you want to continue? (Yes, No)[Yes]:                                                                              
          It has been detected that this program is executed through an SSH connection without using screen.                              
          Continuing with the installation may lead to broken installation if the network connection fails.                               
          It is highly recommended to abort the installation and run it inside a screen session using command "screen".                   
          Do you want to continue anyway? (Yes, No)[No]: yes                                                                              
[ INFO  ] Generating a temporary VNC password.                                                                                            
[ INFO  ] Stage: Environment setup                                                                                                        
          Configuration files: []                                                                                                         
          Log file: /var/log/ovirt-hosted-engine-setup/ovirt-hosted-engine-setup-20141002100154-t5upq8.log                                
          Version: otopi-1.3.0_master (otopi-1.3.0-0.0.1.master.el6ev)                                                                    
[ INFO  ] Hardware supports virtualization                                                                                                
[ INFO  ] Stage: Environment packages setup                                                                                               
[ INFO  ] Stage: Programs detection                                                                                                       
[ INFO  ] Stage: Environment setup                                                                                                        
[ INFO  ] Waiting for VDSM hardware info                                                                                                  
[ INFO  ] Waiting for VDSM hardware info                                                                                                  
[ INFO  ] Waiting for VDSM hardware info                                                                                                  
[ INFO  ] Waiting for VDSM hardware info                                                                                                  
[ INFO  ] Waiting for VDSM hardware info                                                                                                  
[ INFO  ] Generating libvirt-spice certificates                                                                                           
[ INFO  ] Stage: Environment customization                                                                                                
                                                                                                                                          
          --== STORAGE CONFIGURATION ==--                                                                                                 
                                                                                                                                          
          During customization use CTRL-D to abort.                                                                                       
          Please specify the storage you would like to use (iscsi, nfs3, nfs4)[nfs3]:                                                     
          Please specify the full shared storage connection path to use (example: host:/path): 10.35.160.108:/RHEV/nsednev_HE_3_5         
[ INFO  ] Installing on first host                                                                                                        
          Please provide storage domain name. [hosted_storage]:                                                                           
          Local storage datacenter name is an internal name and currently will not be shown in engine's admin UI.Please enter local datacenter name [hosted_datacenter]:                                                                                                                                              
                                                                                                                                                           
          --== SYSTEM CONFIGURATION ==--                                                                                                                   
                                                                                                                                                           
                                                                                                                                                           
          --== NETWORK CONFIGURATION ==--                                                                                                                  
                                                                                                                                                           
          Please indicate a nic to set rhevm bridge on: (eth3, eth2, eth1, eth0) [eth3]: eth0                                                              
          iptables was detected on your computer, do you wish setup to configure it? (Yes, No)[Yes]:                                                       
          Please indicate a pingable gateway IP address [10.35.117.254]:                                                                                   
                                                                                                                                                           
          --== VM CONFIGURATION ==--                                                                                                                       
                                                                                                                                                           
          Please specify the device to boot the VM from (cdrom, disk, pxe) [cdrom]: pxe                                                                    
          The following CPU types are supported by this host:                                                                                              
                 - model_SandyBridge: Intel SandyBridge Family                                                                                             
                 - model_Westmere: Intel Westmere Family                                                                                                   
                 - model_Nehalem: Intel Nehalem Family                                                                                                     
                 - model_Penryn: Intel Penryn Family                                                                                                       
                 - model_Conroe: Intel Conroe Family                                                                                                       
          Please specify the CPU type to be used by the VM [model_SandyBridge]:                                                                            
          Please specify the number of virtual CPUs for the VM [Defaults to minimum requirement: 2]:                                                       
          Please specify the disk size of the VM in GB [Defaults to minimum requirement: 25]:                                                              
          You may specify a unicast MAC address for the VM or accept a randomly generated default [00:16:3e:7b:be:41]: 00:16:3E:7B:B8:53                   
          Please specify the memory size of the VM in MB [Defaults to minimum requirement: 4096]:                                                          
          Please specify the console type you would like to use to connect to the VM (vnc, spice) [vnc]:                                                   
                                                                                                                                                           
          --== HOSTED ENGINE CONFIGURATION ==--                                                                                                            
                                                                                                                                                           
          Enter the name which will be used to identify this host inside the Administrator Portal [hosted_engine_1]:                                       
          Enter 'admin@internal' user password that will be used for accessing the Administrator Portal:                                                   
          Confirm 'admin@internal' user password:                                                                                                          
          Please provide the FQDN for the engine you would like to use.                                                                                    
          This needs to match the FQDN that you will use for the engine installation within the VM.                                                        
          Note: This will be the FQDN of the VM you are now going to create,                                                                               
          it should not point to the base host or to any other existing machine.                                                                           
          Engine FQDN: nsednev-he-1.qa.lab.tlv.redhat.com                                                                                                  
          Please provide the name of the SMTP server through which we will send notifications [localhost]:                                                 
          Please provide the TCP port number of the SMTP server [25]:                                                                                      
          Please provide the email address from which notifications will be sent [root@localhost]:                                                         
          Please provide a comma-separated list of email addresses which will get notifications [root@localhost]:                                          
[ INFO  ] Stage: Setup validation                                                                                                                          
                                                                                                                                                           
          --== CONFIGURATION PREVIEW ==--                                                                                                                  
                                                                                                                                                           
          Bridge interface                   : eth0                                                                                                        
          Engine FQDN                        : nsednev-he-1.qa.lab.tlv.redhat.com                                                                          
          Bridge name                        : rhevm                                                                                                       
          SSH daemon port                    : 22                                                                                                          
          Firewall manager                   : iptables
          Gateway address                    : 10.35.117.254
          Host name for web application      : hosted_engine_1
          Host ID                            : 1
          Image size GB                      : 25
          Storage connection                 : 10.35.160.108:/RHEV/nsednev_HE_3_5
          Console type                       : vnc
          Memory size MB                     : 4096
          MAC address                        : 00:16:3E:7B:B8:53
          Boot type                          : pxe
          Number of CPUs                     : 2
          CPU Type                           : model_SandyBridge

          Please confirm installation settings (Yes, No)[Yes]:
[ INFO  ] Generating answer file '/etc/ovirt-hosted-engine/answers.conf'
[ INFO  ] Stage: Transaction setup
[ INFO  ] Stage: Misc configuration
[ INFO  ] Stage: Package installation
[ INFO  ] Stage: Misc configuration
[ INFO  ] Configuring libvirt
[ INFO  ] Configuring VDSM
[ INFO  ] Starting vdsmd
[ INFO  ] Waiting for VDSM hardware info
[ INFO  ] Waiting for VDSM hardware info
[ INFO  ] Configuring the management bridge
[ INFO  ] Creating Storage Domain
[ INFO  ] Creating Storage Pool
[ INFO  ] Connecting Storage Pool
[ INFO  ] Verifying sanlock lockspace initialization
[ INFO  ] Creating VM Image
[ INFO  ] Disconnecting Storage Pool
[ INFO  ] Start monitoring domain
[ INFO  ] Configuring VM
[ INFO  ] Updating hosted-engine configuration
[ INFO  ] Stage: Transaction commit
[ INFO  ] Stage: Closing up
[ INFO  ] Creating VM
[ ERROR ] Failed to execute stage 'Closing up': Cannot set temporary password for console connection. The VM may not have been created: please check VDSM logs
[ INFO  ] Stage: Clean up
[ INFO  ] Generating answer file '/etc/ovirt-hosted-engine/answers.conf'
[ INFO  ] Answer file '/etc/ovirt-hosted-engine/answers.conf' has been updated
[ INFO  ] Stage: Pre-termination
[ INFO  ] Stage: Termination

Version-Release number of selected component (if applicable):
libvirt-0.10.2-46.el6.x86_64
qemu-kvm-rhev-0.12.1.2-2.448.el6.x86_64
vdsm-4.16.5-2.el6ev.x86_64
sanlock-2.8-1.el6.x86_64
Linux version 2.6.32-502.el6.x86_64 (mockbuild@x86-026.build.eng.bos.redhat.com) (gcc version 4.4.7 20120313 (Red Hat 4.4.7-10) (GCC) ) #1 SMP Mon Sep 8 20:13:50 EDT 2014


How reproducible:
100%

Steps to Reproduce:
1.yum install ovirt-hosted-engine-setup -y
2.hosted-engine --deploy

Actual results:
HE deployment, fails...

Expected results:
HE deployed.

Comment 1 Sandro Bonazzola 2014-10-07 12:25:07 UTC
Can you please attach relevant logs? (vdsm, libvirt, hosted-engine setup)

Comment 2 Nikolai Sednev 2014-10-13 09:20:21 UTC
(In reply to Sandro Bonazzola from comment #1)
> Can you please attach relevant logs? (vdsm, libvirt, hosted-engine setup)

Hi,
Can you run the deployment on RHEL6.6 and see the result, components not were installed it seems, as deployment failed.
Please try to run on your environment.

Comment 3 Nikolai Sednev 2014-10-13 12:24:39 UTC
Oct 13 14:18:09 blue-vdsc yum[7567]: Installed: libvirt-client-0.10.2-46.el6.x86_64
Oct 13 14:18:10 blue-vdsc yum[7567]: Installed: libvirt-python-0.10.2-46.el6.x86_64
Oct 13 14:18:12 blue-vdsc yum[7567]: Installed: libvirt-0.10.2-46.el6.x86_64
Oct 13 14:18:14 blue-vdsc yum[7567]: Installed: libvirt-lock-sanlock-0.10.2-46.el6.x86_64
Oct 13 14:31:12 blue-vdsc kernel: type=1400 audit(1413199872.822:9): avc:  denied  { connectto } for  pid=10791 comm="libvirtd" path="/var/run/sanlock/sanlock.sock" scontext=system_u:system_r:svirt_t:s0:c19,c423 tcontext=unconfined_u:system_r:sanlock_t:s0-s0:c0.c1023 tclass=unix_stream_socket
Oct 13 14:31:12 blue-vdsc kernel: type=1400 audit(1413199872.822:10): avc:  denied  { connectto } for  pid=10791 comm="libvirtd" path="/var/run/sanlock/sanlock.sock" scontext=system_u:system_r:svirt_t:s0:c19,c423 tcontext=unconfined_u:system_r:sanlock_t:s0-s0:c0.c1023 tclass=unix_stream_socket
Oct 13 14:31:12 blue-vdsc kernel: libvirtd[10791]: segfault at fffffffffffffffc ip 00007f6ab56e353c sp 00007f6aaed27938 error 4 in libc-2.12.so[7f6ab5668000+18a000]
Oct 13 14:31:13 blue-vdsc vdsm vm.Vm ERROR vmId=`6b4f2645-b464-4ce2-8c48-bbe220221380`::The vm start process failed#012Traceback (most recent call last):#012  File "/usr/share/vdsm/virt/vm.py", line 2266, in _startUnderlyingVm#012    self._run()#012  File "/usr/share/vdsm/virt/vm.py", line 3368, in _run#012    self._connection.createXML(domxml, flags),#012  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 111, in wrapper#012    ret = f(*args, **kwargs)#012  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 2709, in createXML#012    if ret is None:raise libvirtError('virDomainCreateXML() failed', conn=self)#012libvirtError: Child quit during startup handshake: Input/output error

Comment 4 Nikolai Sednev 2014-10-13 12:26:09 UTC
Created attachment 946357 [details]
logs

Comment 5 Nikolai Sednev 2014-10-13 12:27:59 UTC
Reran and sending some logs.
Components were used:
qemu-kvm-rhev-0.12.1.2-2.448.el6.x86_64
ovirt-hosted-engine-setup-1.2.1-1.el6ev.noarch
libvirt-0.10.2-46.el6.x86_64
ovirt-hosted-engine-ha-1.2.2-2.el6ev.noarch
sanlock-2.8-1.el6.x86_64
vdsm-4.16.6-1.el6ev.x86_64

Comment 6 Sandro Bonazzola 2014-10-14 12:46:43 UTC
Moving to vdsm component. AVC denial on accessing sanlock socket. Not sure if it's a virt or storage issue, starting with virt.

Comment 7 Dan Kenigsberg 2014-10-14 13:24:44 UTC
Nikolai, could you provide a core dump of the segfaulting libvirt (in another bug)?

Oct 13 14:31:12 blue-vdsc kernel: libvirtd[10791]: segfault at fffffffffffffffc ip 00007f6ab56e353c sp 00007f6aaed27938 error 4 in libc-2.12.so[7f6ab5668000+18a000]

may be nastier than a missing policy rule.

Comment 8 Nir Soffer 2014-10-14 13:26:26 UTC
(In reply to Sandro Bonazzola from comment #6)
> Moving to vdsm component. AVC denial on accessing sanlock socket. Not sure
> if it's a virt or storage issue, starting with virt.

This does not look like vdsm issue at all. libvirtd cannot access sanlock socket - that looks like selinux-policy issue.

Nikolai, please open selinux-policy bug, and include there the relevant /var/log/audit.log.

Note that audit.log is rotated quickly. To find the relevant log, use:

xzgrep 'comm="libvirtd" path="/var/run/sanlock/sanlock.sock"' /var/log/audit/audit.log*

If this fails, reproduce again and take the current file /var/log/audit/audit.log

Comment 9 Sandro Bonazzola 2014-10-15 07:33:32 UTC
*** Bug 1150427 has been marked as a duplicate of this bug. ***

Comment 10 Simone Tiraboschi 2014-10-15 07:46:47 UTC
Please cross-check with
https://bugzilla.redhat.com/show_bug.cgi?id=1146529
where it seams that some rules was added for Sanlock on selinux-policy-3.13.1-3.el7

Comment 11 Simone Tiraboschi 2014-10-15 10:35:47 UTC
I tried to setup hosted-engine on RHEL7 with selinux-policy-3.13.1-4 but it's still blocked by selinux.

 time->Wed Oct 15 10:02:37 2014
 type=SYSCALL msg=audit(1413360157.163:3312): arch=c000003e syscall=2 success=no exit=-13 a0=7f2a48966410 a1=105002 a2=0 a3=1 items=0 ppid=1 pid=5195 auid=4294967295 uid=179 gid=179 euid=179 suid=179 fsuid=179 egid=179 sgid=179 fsgid=179 tty=(none) ses=4294967295 comm="sanlock" exe="/usr/sbin/sanlock" subj=system_u:system_r:sanlock_t:s0-s0:c0.c1023 key=(null)
 type=AVC msg=audit(1413360157.163:3312): avc:  denied  { read write } for  pid=5195 comm="sanlock" name="dm-9" dev="devtmpfs" ino=611495 scontext=system_u:system_r:sanlock_t:s0-s0:c0.c1023 tcontext=system_u:object_r:svirt_image_t:s0 tclass=blk_file

I'm not really sure it's a selinux-policy issue or a labeling problem from vdsm.
 [root@r70st1 ~]# ls -lZ /dev/dm-9 
 brw-rw----. vdsm qemu system_u:object_r:svirt_image_t:s0 /dev/dm-9

Comment 12 Nir Soffer 2014-10-15 10:52:40 UTC
(In reply to Simone Tiraboschi from comment #11)
> I tried to setup hosted-engine on RHEL7 with selinux-policy-3.13.1-4 but
> it's still blocked by selinux.
> 
>  time->Wed Oct 15 10:02:37 2014
>  type=SYSCALL msg=audit(1413360157.163:3312): arch=c000003e syscall=2
> success=no exit=-13 a0=7f2a48966410 a1=105002 a2=0 a3=1 items=0 ppid=1
> pid=5195 auid=4294967295 uid=179 gid=179 euid=179 suid=179 fsuid=179
> egid=179 sgid=179 fsgid=179 tty=(none) ses=4294967295 comm="sanlock"
> exe="/usr/sbin/sanlock" subj=system_u:system_r:sanlock_t:s0-s0:c0.c1023
> key=(null)
>  type=AVC msg=audit(1413360157.163:3312): avc:  denied  { read write } for 
> pid=5195 comm="sanlock" name="dm-9" dev="devtmpfs" ino=611495
> scontext=system_u:system_r:sanlock_t:s0-s0:c0.c1023
> tcontext=system_u:object_r:svirt_image_t:s0 tclass=blk_file

This seems to be error when sanlock try to read from /dev/dm-9, and
it does not seems to be related to the libvirtd denial.

> 
> I'm not really sure it's a selinux-policy issue or a labeling problem from
> vdsm.
>  [root@r70st1 ~]# ls -lZ /dev/dm-9 
>  brw-rw----. vdsm qemu system_u:object_r:svirt_image_t:s0 /dev/dm-9

This label works on RHEL 7.0 - not sure if with the selinux policy you tested.

If this label does not work, the vm will pause. This cannot cause libvirtd denial.

To ensure that this is the case, can you try to apply this patch and test again?
http://gerrit.ovirt.org/#/c/33931

Comment 13 Simone Tiraboschi 2014-10-15 13:01:34 UTC
I runned it again updating to vdsm-4.16.7-1.el7 from a yesterday build and now vdsm it's able to start the VM over iscsi with sanlock.

Comment 14 Nir Soffer 2014-10-15 13:07:28 UTC
(In reply to Simone Tiraboschi from comment #13)
> I runned it again updating to vdsm-4.16.7-1.el7 from a yesterday build and
> now vdsm it's able to start the VM over iscsi with sanlock.

So this seem like a duplicate of bug 1127460, or it should depend on it.

Comment 15 Simone Tiraboschi 2014-10-15 13:13:30 UTC
(In reply to Nir Soffer from comment #14)
> (In reply to Simone Tiraboschi from comment #13)
> > I runned it again updating to vdsm-4.16.7-1.el7 from a yesterday build and
> > now vdsm it's able to start the VM over iscsi with sanlock.
> 
> So this seem like a duplicate of bug 1127460, or it should depend on it.

Yes, it's not strictly a duplicate but they could be solved as one.

Comment 16 Nir Soffer 2014-10-15 13:29:49 UTC
(In reply to Simone Tiraboschi from comment #15)
> (In reply to Nir Soffer from comment #14)
> > (In reply to Simone Tiraboschi from comment #13)
> > > I runned it again updating to vdsm-4.16.7-1.el7 from a yesterday build and
> > > now vdsm it's able to start the VM over iscsi with sanlock.
> > 
> > So this seem like a duplicate of bug 1127460, or it should depend on it.
> 
> Yes, it's not strictly a duplicate but they could be solved as one.

Why an update was needed to get latest vdsm - hosted engine setup does not require this version? Maybe the spec should be updated?

Comment 17 Simone Tiraboschi 2014-10-15 13:55:56 UTC
that hosted-engine simply requires vdsm >= 4.16.6
while this has been addressed by vdsm-4.16.7-1.el7 which isn't still officially released, so it's just a matter of time.

Comment 18 Nikolai Sednev 2014-10-22 12:05:43 UTC
Hi,
Please provide fixed in version field contents and as for current system behaviour, I was able to deploy HE with 3.5, while selinux was set to 0 "setenforce 0" on host running RHEL6.6.

Comment 19 Michal Skrivanek 2014-11-05 07:36:19 UTC
proper selinux packages has been released on 6.6.z


Note You need to log in before you can comment on or make changes to this bug.