Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 981589 - quota: bricks down with crash
Summary: quota: bricks down with crash
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Gluster Storage
Classification: Red Hat
Component: glusterd
Version: 2.1
Hardware: x86_64
OS: Linux
high
high
Target Milestone: ---
: ---
Assignee: Raghavendra Bhat
QA Contact: Sudhir D
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2013-07-05 08:24 UTC by Saurabh
Modified: 2016-01-19 06:12 UTC (History)
6 users (show)

Fixed In Version: v3.4.0.12rhs.beta4
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2013-09-23 22:39:54 UTC
Target Upstream Version:


Attachments (Terms of Use)

Description Saurabh 2013-07-05 08:24:15 UTC
Description of problem:
bricks go down with a crash seen for them, while putting quota limit on a directory which is having I/O over nfs mount,

[root@quota1 ~]# gluster volume info
 
Volume Name: dist-rep
Type: Distributed-Replicate
Volume ID: b1b80b68-b98b-4aab-a563-3c386c39b842
Status: Started
Number of Bricks: 6 x 2 = 12
Transport-type: tcp
Bricks:
Brick1: 10.70.37.98:/rhs/bricks/d1r1
Brick2: 10.70.37.174:/rhs/bricks/d1r2
Brick3: 10.70.37.136:/rhs/bricks/d2r1
Brick4: 10.70.37.168:/rhs/bricks/d2r2
Brick5: 10.70.37.98:/rhs/bricks/d3r1
Brick6: 10.70.37.174:/rhs/bricks/d3r2
Brick7: 10.70.37.136:/rhs/bricks/d4r1
Brick8: 10.70.37.168:/rhs/bricks/d4r2
Brick9: 10.70.37.98:/rhs/bricks/d5r1
Brick10: 10.70.37.174:/rhs/bricks/d5r2
Brick11: 10.70.37.136:/rhs/bricks/d6r1
Brick12: 10.70.37.168:/rhs/bricks/d6r2
Options Reconfigured:
features.quota: on
[root@quota1 ~]# 
[root@quota1 ~]# 
[root@quota1 ~]# gluster volume status
Status of volume: dist-rep
Gluster process                                         Port    Online  Pid
------------------------------------------------------------------------------
Brick 10.70.37.98:/rhs/bricks/d1r1                      49152   Y       25494
Brick 10.70.37.174:/rhs/bricks/d1r2                     49152   Y       26755
Brick 10.70.37.136:/rhs/bricks/d2r1                     49152   Y       27137
Brick 10.70.37.168:/rhs/bricks/d2r2                     49152   Y       16599
Brick 10.70.37.98:/rhs/bricks/d3r1                      N/A     N       25505
Brick 10.70.37.174:/rhs/bricks/d3r2                     49153   Y       26766
Brick 10.70.37.136:/rhs/bricks/d4r1                     49153   Y       27148
Brick 10.70.37.168:/rhs/bricks/d4r2                     49153   Y       16610
Brick 10.70.37.98:/rhs/bricks/d5r1                      49154   Y       25516
Brick 10.70.37.174:/rhs/bricks/d5r2                     N/A     N       26777
Brick 10.70.37.136:/rhs/bricks/d6r1                     49154   Y       27159
Brick 10.70.37.168:/rhs/bricks/d6r2                     49154   Y       16621
NFS Server on localhost                                 2049    Y       25529
Self-heal Daemon on localhost                           N/A     Y       25537
NFS Server on d6203fcb-6e40-417e-986b-6a74c7f23c42      2049    Y       16641
Self-heal Daemon on d6203fcb-6e40-417e-986b-6a74c7f23c4
2                                                       N/A     Y       16655
NFS Server on cc7bc8ba-fa3a-43d9-a899-114e34d27eb4      2049    Y       26800
Self-heal Daemon on cc7bc8ba-fa3a-43d9-a899-114e34d27eb
4                                                       N/A     Y       26809
NFS Server on 1eb725d7-70d7-4075-a580-b9711fc7affd      2049    Y       27175
Self-heal Daemon on 1eb725d7-70d7-4075-a580-b9711fc7aff
d                                                       N/A     Y       27182
 
There are no active volume tasks
[root@quota1 ~]# 


Version-Release number of selected component (if applicable):
[root@quota1 ~]# rpm -qa | grep glusterfs
glusterfs-fuse-3.4.0.12rhs.beta2-1.el6rhs.x86_64
glusterfs-server-3.4.0.12rhs.beta2-1.el6rhs.x86_64
glusterfs-3.4.0.12rhs.beta2-1.el6rhs.x86_64
[root@quota1 ~]# 

How reproducible:
seen for the first time

Steps to Reproduce:
1. create a volume, start it
2. set quota on the root of the volume
3. mount it over nfs
4. start linux untar and put quota on the directory on which I/O is happening,
in this case,
[root@quota1 ~]# gluster volume quota dist-rep list
                  Path                   Hard-limit Soft-limit   Used  Available
--------------------------------------------------------------------------------
/                                           20GB       90%      10.7MB  20.0GB
/linux-2.6.31.1/Documentation                5GB       90%       1.7MB   5.0GB
[root@quota1 ~]# 


also, infact try to modify the limits for root and directory in consideration to "higher value" than earlier.

Actual results:

2013-07-05 00:48:48.373150] W [common-utils.c:841:gf_string2percent] (-->/usr/lib64/libglusterfs.so.0(+0x5376a) [0x7fc93fcde76a] (-->/usr/lib64/glusterfs/3.4.0.12rhs.beta2/xlator/features/quota.so(reconfigure+0x13d) [0x7fc93a3f1abd] (-->/usr/lib64/glusterfs/3.4.0.12rhs.beta2/xlator/features/quota.so(quota_parse_limits+0x1c8) [0x7fc93a3f1668]))) 0-dist-rep-quota: argument invalid
pending frames:
frame : type(0) op(0)
frame : type(0) op(0)

patchset: git://git.gluster.com/glusterfs.git
signal received: 11
time of crash: 2013-07-05 00:48:48configuration details:
argp 1
backtrace 1
dlfcn 1
fdatasync 1
libpthread 1
llistxattr 1
setfsid 1
spinlock 1
epoll.h 1
xattr.h 1
st_atim.tv_nsec 1
package-string: glusterfs 3.4.0.12rhs.beta2
/lib64/libc.so.6[0x325e432920]
/lib64/libc.so.6[0x325e478641]
/usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x129)[0x7fc93fca4989]
/usr/lib64/libglusterfs.so.0(gf_string2percent+0x111)[0x7fc93fcb1af1]
/usr/lib64/glusterfs/3.4.0.12rhs.beta2/xlator/features/quota.so(quota_parse_limits+0x1c8)[0x7fc93a3f1668]
/usr/lib64/glusterfs/3.4.0.12rhs.beta2/xlator/features/quota.so(reconfigure+0x13d)[0x7fc93a3f1abd]
/usr/lib64/libglusterfs.so.0(+0x5376a)[0x7fc93fcde76a]
/usr/lib64/libglusterfs.so.0(+0x536fc)[0x7fc93fcde6fc]
---------
(END) 

Expected results:

quota limit being getting modified or getting added while I/O is happening is a very general scenario, so bricks going down with crash is not acceptable.

Additional info:

Comment 5 Sachidananda Urs 2013-07-18 11:17:55 UTC
If this is fixed in 18th July 2013 build, please move the status to appropriate state.

Comment 7 Scott Haines 2013-09-23 22:39:54 UTC
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. 

For information on the advisory, and where to find the updated files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2013-1262.html

Comment 8 Scott Haines 2013-09-23 22:43:49 UTC
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. 

For information on the advisory, and where to find the updated files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2013-1262.html


Note You need to log in before you can comment on or make changes to this bug.