Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 1104083 - [rhevh] No VM's core dumps because of missing SELinux policy
Summary: [rhevh] No VM's core dumps because of missing SELinux policy
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Virtualization Manager
Classification: Red Hat
Component: ovirt-node
Version: 3.4.0
Hardware: Unspecified
OS: Unspecified
urgent
urgent
Target Milestone: ---
: 3.5.0
Assignee: Fabian Deutsch
QA Contact: Virtualization Bugs
URL:
Whiteboard: node
Depends On:
Blocks: 1118297 1120065 1123329 rhev3.5beta 1156165
TreeView+ depends on / blocked
 
Reported: 2014-06-03 09:05 UTC by Jiri Belka
Modified: 2016-02-10 20:07 UTC (History)
16 users (show)

Fixed In Version: rhev-hypervisor6-6.6-20141218.0.iso
Doc Type: Bug Fix
Doc Text:
Clone Of:
: 1120065 (view as bug list)
Environment:
Last Closed: 2015-02-11 20:58:41 UTC
oVirt Team: Node
Target Upstream Version:


Attachments (Terms of Use)
SElinux.tar.gz (deleted)
2015-01-15 10:34 UTC, cshao
no flags Details


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHEA-2015:0160 normal SHIPPED_LIVE ovirt-node bug fix and enhancement update 2015-02-12 01:34:52 UTC
oVirt gerrit 28344 master MERGED selinux: Fix VM core dumps Never
Red Hat Bugzilla 1215554 None None None Never

Internal Links: 1215554

Description Jiri Belka 2014-06-03 09:05:55 UTC
Description of problem:
No VMs core dumps because of missing SELinux local policies.

type=AVC msg=audit(1401786061.559:42800): avc:  denied  { getattr } for  pid=10087 comm="logrotate" path="/var/log/core/core.9784.1401785994.dump" dev=dm-9 ino=3670018 scont
ext=system_u:system_r:logrotate_t:s0-s0:c0.c1023 tcontext=system_u:object_r:virt_cache_t:s0:c462,c953 tclass=file

# cat /etc/redhat-release 
Red Hat Enterprise Virtualization Hypervisor release 6.5 (20140527.0.el6ev)
[root@dell-r210ii-04 ~]# semanage fcontext -l | grep /var/log/core
[root@dell-r210ii-04 ~]#

# cat /etc/redhat-release 
Red Hat Enterprise Linux Server release 6.5 (Santiago)
[root@srh-03 ~]# semanage fcontext -l  |grep /var/log/core
/var/log/core(/.*)?                                all files          system_u:object_r:virt_cache_t:s0 
[root@srh-03 ~]#

On normal RHEL qemu is able to write into 'virt_cache_t' dirs. On RHEVH it cannot. 'semanage fcontext -a -t virt_cache_t "/var/log/core(./*)?"' solves the problem.

Without core dumps one is not able to debug VM crashes, like recent VMs' craches because of spice - BZ1092380.

Version-Release number of selected component (if applicable):
selinux-policy-3.7.19-231.el6_5.3.noarch
selinux-policy-targeted-3.7.19-231.el6_5.3.noarch
vdsm-python-4.14.7-3.el6ev.x86_64
vdsm-python-zombiereaper-4.14.7-3.el6ev.noarch
vdsm-xmlrpc-4.14.7-3.el6ev.noarch
vdsm-4.14.7-3.el6ev.x86_64
vdsm-reg-4.14.7-3.el6ev.noarch
vdsm-hook-vhostmd-4.14.7-3.el6ev.noarch
vdsm-cli-4.14.7-3.el6ev.noarch

How reproducible:
100%

Steps to Reproduce:
1. pkill -ABRT -f 'qemu-kvm'
2.
3.

Actual results:
no core dump

Expected results:
VMs core dumps available

Additional info:

Comment 1 Jiri Belka 2014-06-03 09:08:15 UTC
Ah, relevant bug about core dumping qemu processes (because of spice) is in fact BZ1072101, not the BZ written in #0 (cut&paste issue).

Comment 2 Fabian Deutsch 2014-06-03 16:46:55 UTC
(In reply to Jiri Belka from comment #0)
> Description of problem:
> No VMs core dumps because of missing SELinux local policies.
> 
> type=AVC msg=audit(1401786061.559:42800): avc:  denied  { getattr } for 
> pid=10087 comm="logrotate" path="/var/log/core/core.9784.1401785994.dump"
> dev=dm-9 ino=3670018 scont
> ext=system_u:system_r:logrotate_t:s0-s0:c0.c1023
> tcontext=system_u:object_r:virt_cache_t:s0:c462,c953 tclass=file
> 
> # cat /etc/redhat-release 
> Red Hat Enterprise Virtualization Hypervisor release 6.5 (20140527.0.el6ev)
> [root@dell-r210ii-04 ~]# semanage fcontext -l | grep /var/log/core
> [root@dell-r210ii-04 ~]#
> 
> # cat /etc/redhat-release 
> Red Hat Enterprise Linux Server release 6.5 (Santiago)
> [root@srh-03 ~]# semanage fcontext -l  |grep /var/log/core
> /var/log/core(/.*)?                                all files         
> system_u:object_r:virt_cache_t:s0 
> [root@srh-03 ~]#

Hey Jiri,

the paste above looks like you were using RHEL.

Were you using RHEV-H and if - please tell me which RHEV-H and which ovirt-node version you were using.

Comment 3 Jiri Belka 2014-06-04 06:32:56 UTC
#0 contains outputs from RHEVH and RHEL for comparison.

# cat /etc/redhat-release 
Red Hat Enterprise Virtualization Hypervisor release 6.5 (20140527.0.el6ev)

# rpm -qa | egrep "ovirt|redhat-release"
redhat-release-server-6Server-6.5.0.2.el6_5.x86_64
ovirt-host-deploy-offline-1.2.1-1.el6ev.x86_64
ovirt-node-plugin-cim-3.0.1-18.el6_5.10.noarch
ovirt-node-3.0.1-18.el6_5.10.noarch
ovirt-node-plugin-rhn-3.0.1-18.el6_5.10.noarch
ovirt-node-plugin-snmp-3.0.1-18.el6_5.10.noarch
ovirt-node-selinux-3.0.1-18.el6_5.10.noarch
ovirt-node-plugin-vdsm-0.1.1-22.el6ev.noarch

Comment 4 Fabian Deutsch 2014-06-04 07:43:23 UTC
Hey Jiri,

thanks.

Comment 5 cshao 2014-07-10 10:05:44 UTC
RHEV-H QE can reproduce this bug.

Test version:
rhev-hypervisor6-6.5-20140624.0.el6ev
ovirt-node-3.0.1-18.el6_5.11.noarch
vdsm-4.14.7-3.el6ev.x86_64
RHEVM av10
selinux-policy-3.7.19-231.el6_5.3.noarch
selinux-policy-targeted-3.7.19-231.el6_5.3.noarch
vdsm-python-4.14.7-3.el6ev.x86_64
vdsm-python-zombiereaper-4.14.7-3.el6ev.noarch
vdsm-xmlrpc-4.14.7-3.el6ev.noarch
vdsm-4.14.7-3.el6ev.x86_64
vdsm-reg-4.14.7-3.el6ev.noarch
vdsm-hook-vhostmd-4.14.7-3.el6ev.noarch
vdsm-cli-4.14.7-3.el6ev.noarch

# cat /etc/redhat-release
Red Hat Enterprise Virtualization Hypervisor release 6.5 (20140624.0.el6ev) (Edited)
[root@localhost admin]# semanage fcontext -l | grep /var/log/core

# cat /etc/redhat-release 
Red Hat Enterprise Linux Server release 6.5 (Santiago)
[root@dell-op740-03 tmp]# semanage fcontext -l  |grep /var/log/core
/var/log/core(/.*)?                                all files          system_u:object_r:virt_cache_t:s0 

Test steps:
1. Install RHEV-H and register to RHEVM.
2. Create VM.
3. Run command "pkill -ABRT -f 'qemu-kvm'" on host.

Test result:
no core dump

Comment 6 cshao 2014-07-10 11:37:53 UTC
Hi Jbelka, 

I append 'semanage fcontext -a -t virt_cache_t "/var/log/core(./*)?"' to RHEV-H host, and then run command "pkill -ABRT -f 'qemu-kvm'",
but there is still no core dump in vm.

Can you point me which step was wrong?

Test steps:
1. Install RHEV-H and register to RHEVM.
2. Create VM and config kdump to local(/data/core)
3. Run workaround: 'semanage fcontext -a -t virt_cache_t "/var/log/core(./*)?"' to RHEV-H host, 
4. Run command "pkill -ABRT -f 'qemu-kvm'" on host.
5. Start the VM again, check the core dump file under(/data/core)

Test result:
step4, all VMs were killed.
step5, no core dump.

Thanks!

Comment 7 Jiri Belka 2014-07-10 12:17:13 UTC
/data/core is probably not correctly labeled. See audit.log for denials.

Comment 8 cshao 2014-07-10 13:12:29 UTC
Finally the core dump file can generate successful.

Details workaround is here:
#'semanage fcontext -a -t virt_cache_t "/var/log/core(./*)?"
#restorecon -RFv /var/log/core

#ls -ldZ /data/core /var/log/core
drwxrwxrwt. root root system_u:object_r:virt_cache_t:s0 /data/core
drwxrwxrwt. root root system_u:object_r:virt_cache_t:s0 /var/log/core

Comment 11 cshao 2015-01-14 07:37:30 UTC
Test version:
rhev-hypervisor6-6.6-20150112.0
ovirt-node-3.1.0-0.42.20150109gitd06b7c5.el6.noarch
selinux-policy-3.7.19-260.el6_6.1.noarch
selinux-policy-targeted-3.7.19-260.el6_6.1.noarch

Test steps:
1. Install RHEV-H and register to RHEVM.
2. Create VM.
3. Run command "pkill -ABRT -f 'qemu-kvm'" on host.
4. Run  ll /var/log/core/ on host.

Test result:
VMs core dumps available

# ll /var/log/core/
total 959676
-rw-------. 1 qemu qemu 1302220800 2015-01-14 07:23 core.21219.1421220217.dump


So the bug was fixed with above rhevh6.0 build.

===================================================
RHEVH7.0
rhev-hypervisor7-7.0-20150112.0
ovirt-node-3.1.0-0.42.20150109gitd06b7c5.el7.noarch
selinux-policy-3.12.1-153.el7_0.13.noarch
selinux-policy-targeted-3.12.1-153.el7_0.13.noarch

Still no VM's core dumps in RHEV-H 7.0, 

I noticed that the bug fixed in version: rhev-hypervisor6-6.6-20141218.0.iso rhev-hypervisor7-7.0-20141218.0.iso. 

So can I verify this bug directly according the test result of rhevh6.6? 
Or assigned this bug until RHEVH7.0 fix this issue?

Comment 14 cshao 2015-01-15 10:34:50 UTC
Created attachment 980419 [details]
SElinux.tar.gz

Comment 15 Fabian Deutsch 2015-01-15 11:40:41 UTC
Chen, does the core dump work in permissive mode? So booting with enforcing=0

Comment 16 cshao 2015-01-16 14:27:59 UTC
(In reply to Fabian Deutsch from comment #15)
> Chen, does the core dump work in permissive mode? So booting with enforcing=0

Test version:
Red Hat Enterprise Virtualization Hypervisor release 7.0 (20150114.0.2.el7ev)
ovirt-node-3.2.1-1.el7.noarch
selinux-policy-targeted-3.12.1-153.el7_0.13.noarch
selinux-policy-3.12.1-153.el7_0.13.noarch

# cat /proc/cmdline 
BOOT_IMAGE=/vmlinuz0 root=live:LABEL=Root ro rootfstype=auto rootflags=ro rd.live.image rd.live.check crashkernel=256M elevator=deadline quiet max_loop=256 rhgb rd.luks=0 rd.md=0 rd.dm=0 nomodeset enforcing=0

Test steps:
1. Install RHEV-H add booting with enforcing=0 
2. Register to RHEVM.
3. Create VM.
4. Run command "pkill -ABRT -f 'qemu-kvm'" on host.
5. Run  ll /var/log/core/ on host.

Test result:
Still no VM's core dumps in RHEV-H 7.0,

Comment 17 Fabian Deutsch 2015-01-16 14:45:59 UTC
Okay, then it is not an selinux issue - and thus it can not be said if this bug is fixed or not, thus I am setting it back to ON_QA

Miroslav, can you tell how we can debug this problem?

Comment 18 Miroslav Rezanina 2015-01-19 09:45:47 UTC
I check the issue with Chen and it looks like ulimit problem. Waiting for test results.

Comment 20 Miroslav Rezanina 2015-01-20 06:15:23 UTC
Do we need ABRT for dump? I can't see it in the system.

Comment 21 Fabian Deutsch 2015-01-21 17:13:55 UTC
No, ABRT is not needed.
But the change to security.conf had no immediate effect in comment 19, that's why the dump did not work.

Setting ulimit -S -c unlimited enables core dumps.

Steps to reproduce:
1. Login as root
2. ulimit -S -c unlimited
3. /usr/libexec/qemu-kvm &
4. pkill -ABRT qemu-kvm
5. ls -shal -var/log/core/

5 Will show a core dump


But note that ulimits for services are set elsewhere.
Does this bug also happen on RHEL?

Comment 22 Fabian Deutsch 2015-01-22 05:46:45 UTC
On RHEL both, soft and hard limits are set to unlimited. On RHEV-H the soft limit is set to 0.

Comment 23 Fabian Deutsch 2015-01-22 05:52:40 UTC
(In reply to Fabian Deutsch from comment #22)
> On RHEL both, soft and hard limits are set to unlimited. On RHEV-H the soft
> limit is set to 0.

That was not correct. By default it is the same as on RHEV-H.

Chen, can you reproduce this bug on RHEL 7 as well?

Comment 29 Israel Pinto 2015-01-22 15:56:30 UTC
Verify it with RHEL 6.6 and 7, on RHEL 6.6 I got core dump but on RHEL 7 the process killed and the core dump did not created.
outputs:
RHEL 6.6:
[root@dingo-vdsc ~]# ps -ef | grep qemu
qemu     14748     1 34 14:07 ?        00:05:06 /usr/libexec/qemu-kvm -name VM-6.6_test -S -M rhel6.5.0 -cpu Penryn -enable-kvm -m 1024 -realtime mlock=off -smp 1,maxcpus=16,sockets=16,cores=1,threads=1 -uuid 45a46df9-166d-42cc-
...
...
[root@dingo-vdsc ~]# pkill -ABRT -f 'qemu'
[root@dingo-vdsc ~]# ps -ef | grep qemu
qemu     16336 14896  0 Jan20 ?        00:00:00 [supervdsmServer] <defunct>
root     16839 16588  0 14:22 pts/0    00:00:00 grep qemu
[root@dingo-vdsc ~]# ll /var/log/core/
total 1380304
-rw-------. 1 qemu qemu 1618182144 Jan 22 14:22 core.14748.1421929347.dump
-rw-------. 1 vdsm kvm   189968384 Dec  5 14:49 core.9864.1417783798.dump
[root@dingo-vdsc ~]# date
Thu Jan 22 14:23:23 IST 2015
[root@dingo-vdsc ~]#

RHEL 7:
[root@puma18 ~]# ps -ef | grep qemu
qemu     18122     1 15 14:31 ?        00:00:19 /usr/libexec/qemu-kvm -name VM-7_test -S -machine rhel6.5.0,accel=kvm,usb=off -cpu Penryn -m 1024 -realtime mlock=off -smp 1,maxcpus=16,sockets=16,cores=1,threads=1 -uuid
...
...
[root@puma18 ~]# pkill -ABRT -f 'qemu'
[root@puma18 ~]# ll /var/log/core/
total 0
[root@puma18 ~]# ps -ef | grep qemu
qemu     15096 14849  0 14:07 ?        00:00:00 [supervdsmServer] <defunct>
root     18531 17658  0 14:34 pts/0    00:00:00 grep --color=auto qemu

Comment 32 errata-xmlrpc 2015-02-11 20:58:41 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHEA-2015-0160.html


Note You need to log in before you can comment on or make changes to this bug.