Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 1360923 - [ceph-ansible]: purge-cluster fails in one task and do not clean up osds
Summary: [ceph-ansible]: purge-cluster fails in one task and do not clean up osds
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Red Hat Storage Console
Classification: Red Hat
Component: ceph-ansible
Version: 2
Hardware: x86_64
OS: Linux
unspecified
medium
Target Milestone: ---
: 3
Assignee: Gregory Meno
QA Contact: ceph-qe-bugs
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-07-27 20:48 UTC by Rachana Patel
Modified: 2017-05-01 14:49 UTC (History)
7 users (show)

Fixed In Version: ceph-ansible-2.1.9-1.el7scon
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2017-05-01 14:49:18 UTC


Attachments (Terms of Use)

Description Rachana Patel 2016-07-27 20:48:29 UTC
Description of problem:
=========================
purge-cluster fails for task - check for anything running ceph, and do not clean up OSD node


Version-Release number of selected component (if applicable):


How reproducible:
=================
always


Steps to Reproduce:
===================
1. create cluster with one MON and 3 OSD nodes
[ubuntu@magna008 ~]$ sudo ceph osd tree
ID WEIGHT  TYPE NAME         UP/DOWN REWEIGHT PRIMARY-AFFINITY 
-1 5.45485 root default                                        
-2 1.81828     host magna078                                   
 0 0.90919         osd.0          up  1.00000          1.00000 
 3 0.90909         osd.3          up  1.00000          1.00000 
-3 1.81828     host magna051                                   
 2 0.90919         osd.2          up  1.00000          1.00000 
 4 0.90909         osd.4          up  1.00000          1.00000 
-4 1.81828     host magna057                                   
 1 0.90919         osd.1          up  1.00000          1.00000 
 5 0.90909         osd.5          up  1.00000          1.00000 


2. Do some I/O
3. purge cluster.

Actual results:
================
<snippet of run>
TASK: [check for anything running ceph] *************************************** 
failed: [magna057] => {"changed": true, "cmd": "ps awux | grep -v grep | grep -q -- ceph-", "delta": "0:00:00.025591", "end": "2016-07-27 15:38:19.434596", "failed": true, "failed_when_result": true, "rc": 0, "start": "2016-07-27 15:38:19.409005", "stdout_lines": [], "warnings": []}
failed: [magna051] => {"changed": true, "cmd": "ps awux | grep -v grep | grep -q -- ceph-", "delta": "0:00:00.025801", "end": "2016-07-27 15:38:21.278102", "failed": true, "failed_when_result": true, "rc": 0, "start": "2016-07-27 15:38:21.252301", "stdout_lines": [], "warnings": []}
changed: [magna008] => {"changed": true, "cmd": "ps awux | grep -v grep | grep -q -- ceph-", "delta": "0:00:00.025990", "end": "2016-07-27 15:38:24.301096", "failed": false, "failed_when_result": false, "rc": 1, "start": "2016-07-27 15:38:24.275106", "stderr": "", "stdout": "", "stdout_lines": [], "warnings": []}
failed: [magna078] => {"changed": true, "cmd": "ps awux | grep -v grep | grep -q -- ceph-", "delta": "0:00:00.025638", "end": "2016-07-27 15:38:23.198611", "failed": true, "failed_when_result": true, "rc": 0, "start": "2016-07-27 15:38:23.172973", "stdout_lines": [], "warnings": []}




OSD node:-
osd node
=========
[root@magna051 ubuntu]# ls -l /var/*/ceph
/var/lib/ceph:
total 20
drwxr-x---. 2 ceph ceph 4096 Jul 25 21:48 bootstrap-mds
drwxr-xr-x. 2 ceph ceph 4096 Jul 27 15:32 bootstrap-osd
drwxr-x---. 2 ceph ceph 4096 Jul 25 21:48 bootstrap-rgw
drwxr-x---. 4 ceph ceph 4096 Jul 27 15:33 osd
drwxr-x---. 2 ceph ceph 4096 Jul 27 15:33 tmp

/var/log/ceph:
total 528
-rw-r--r--. 1 root root    864 Jul 27 15:33 ceph-osd.0.log
-rw-r--r--. 1 ceph ceph 267657 Jul 27 15:38 ceph-osd.2.log
-rw-r--r--. 1 ceph ceph 254730 Jul 27 15:38 ceph-osd.4.log
-rw-r--r--. 1 root root   7699 Jul 27 15:33 ceph-osd.admin.log
-rw-r--r--. 1 root root      0 Jul 27 15:33 qemu-guest-27097.log
-rw-r--r--. 1 root root      0 Jul 27 15:33 qemu-guest-27125.log
-rw-r--r--. 1 root root      0 Jul 27 15:33 qemu-guest-27233.log
-rw-r--r--. 1 root root      0 Jul 27 15:33 qemu-guest-27741.log
-rw-r--r--. 1 root root      0 Jul 27 15:33 qemu-guest-27770.log
-rw-r--r--. 1 root root      0 Jul 27 15:33 qemu-guest-27896.log
-rw-r--r--. 1 root root      0 Jul 27 15:33 qemu-guest-27964.log

/var/run/ceph:
total 0
[root@magna051 ubuntu]# df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1       917G  2.7G  868G   1% /
devtmpfs         16G     0   16G   0% /dev
tmpfs            16G     0   16G   0% /dev/shm
tmpfs            16G   17M   16G   1% /run
tmpfs            16G     0   16G   0% /sys/fs/cgroup
tmpfs           3.2G     0  3.2G   0% /run/user/1000
/dev/sdb1       932G   34M  932G   1% /var/lib/ceph/osd/ceph-2
/dev/sdc1       931G   34M  931G   1% /var/lib/ceph/osd/ceph-4




Expected results:
==================
It should clean up all OSD node

Additional info:

Comment 6 Ken Dreyer (Red Hat) 2017-03-03 17:07:26 UTC
We think this is likely fixed in the latest ceph-ansible RPM going through testing (v2.1.9 as of this writing.) Would you please retest with the latest version?


Note You need to log in before you can comment on or make changes to this bug.