Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 1364329 - Glusterd crashes upon receiving SIGUSR1
Summary: Glusterd crashes upon receiving SIGUSR1
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: GlusterFS
Classification: Community
Component: glusterd
Version: 3.8.2
Hardware: x86_64
OS: Linux
unspecified
high
Target Milestone: ---
Assignee: Atin Mukherjee
QA Contact:
URL:
Whiteboard:
: 1359131 (view as bug list)
Depends On: 1359131 1359190
Blocks: 1364331
TreeView+ depends on / blocked
 
Reported: 2016-08-05 05:50 UTC by Atin Mukherjee
Modified: 2016-08-12 09:48 UTC (History)
3 users (show)

Fixed In Version: glusterfs-3.8.2
Doc Type: If docs needed, set a value
Doc Text:
Clone Of: 1359190
: 1364331 (view as bug list)
Environment:
Last Closed: 2016-08-12 09:48:35 UTC
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:


Attachments (Terms of Use)

Description Atin Mukherjee 2016-08-05 05:50:39 UTC
+++ This bug was initially created as a clone of Bug #1359190 +++

+++ This bug was initially created as a clone of Bug #1359131 +++

Description of problem:
I try to generate glusterdump file with SIGUSR1 to find out maximum op-version for the glusterd on this host.
Glusterd crashes upon receiving SIGUSR1


Version-Release number of selected component (if applicable):
# cat /etc/lsb-release 
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=14.04
DISTRIB_CODENAME=trusty
DISTRIB_DESCRIPTION="Ubuntu 14.04.4 LTS"

# dpkg -l | grep gluster
ii  glusterfs-client                      3.8.1-ubuntu1~trusty1               amd64        clustered file-system (client package)
ii  glusterfs-common                      3.8.1-ubuntu1~trusty1               amd64        GlusterFS common libraries and translator modules
ii  glusterfs-server                      3.8.1-ubuntu1~trusty1               amd64        clustered file-system (server package)


How reproducible:
Glusterd crashes each time I send SIGUSR1


Steps to Reproduce:
1. rm -f /var/run/gluster/glusterdump.*
2. service glusterfs-server start
3. kill -SIGUSR1 $(pidof glusterd)
4. ps -ef | grep glusterd ; ls /var/run/gluster | grep glusterdump

Actual results:
No glusterd in the ps output.
No glusterdump file created.

glusterd crashes with the following lines in the log:
[2016-07-22 11:22:50.923915] I [MSGID: 106493] [glusterd-rpc-ops.c:691:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: c6654b8f-3ca8-424f-8d0f-7c0b7d8108cc
pending frames:
patchset: git://git.gluster.com/glusterfs.git
signal received: 11
time of crash: 
2016-07-22 11:23:09
configuration details:
argp 1
backtrace 1
dlfcn 1
libpthread 1
llistxattr 1
setfsid 1
spinlock 1
epoll.h 1
xattr.h 1
st_atim.tv_nsec 1
package-string: glusterfs 3.8.1
/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(_gf_msg_backtrace_nomem+0x92)[0x7f21f4b87b02]
/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(gf_print_trace+0x31d)[0x7f21f4b9104d]
/lib/x86_64-linux-gnu/libc.so.6(+0x36cb0)[0x7f21f3f81cb0]
/lib/x86_64-linux-gnu/libc.so.6(_IO_vfprintf+0x1d03)[0x7f21f3f96943]
/lib/x86_64-linux-gnu/libc.so.6(__vsnprintf_chk+0x95)[0x7f21f4054165]
/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(gf_proc_dump_add_section_fd+0x69)[0x7f21f4bb18a9]
/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(gf_proc_dump_add_section+0xa5)[0x7f21f4bb1ab5]
/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(+0x4ce3e)[0x7f21f4bb1e3e]
/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(gf_proc_dump_xlator_info+0x58)[0x7f21f4bb2658]
/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(gf_proc_dump_info+0x500)[0x7f21f4bb2d00]
/usr/sbin/glusterd(glusterfs_sigwaiter+0xce)[0x55ffe99d2bae]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x8184)[0x7f21f4318184]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7f21f404537d]
---------



Expected results:
Glusterd creates glusterdump file in /var/run/gluster


Additional info:

--- Additional comment from Atin Mukherjee on 2016-07-22 08:47:08 EDT ---

Does this happen everytime? I've seen it earlier as well but couldn't reproduce.

--- Additional comment from Atin Mukherjee on 2016-07-22 08:48:06 EDT ---

coredump file?

--- Additional comment from Vijay Bellur on 2016-07-22 09:10:07 EDT ---

REVIEW: http://review.gluster.org/14987 (libglusterfs: fix glusterd statedump crash) posted (#1) for review on master by Atin Mukherjee (amukherj@redhat.com)

--- Additional comment from Vijay Bellur on 2016-07-22 09:16:55 EDT ---

REVIEW: http://review.gluster.org/14987 (libglusterfs: fix glusterd statedump crash) posted (#2) for review on master by Atin Mukherjee (amukherj@redhat.com)

--- Additional comment from Vijay Bellur on 2016-07-24 23:56:40 EDT ---

REVIEW: http://review.gluster.org/14987 (libglusterfs: fix glusterd statedump crash) posted (#3) for review on master by Atin Mukherjee (amukherj@redhat.com)

--- Additional comment from Vijay Bellur on 2016-07-26 14:10:38 EDT ---

REVIEW: http://review.gluster.org/14987 (libglusterfs: fix glusterd statedump crash) posted (#4) for review on master by Atin Mukherjee (amukherj@redhat.com)

--- Additional comment from Vijay Bellur on 2016-07-29 00:57:17 EDT ---

REVIEW: http://review.gluster.org/14987 (libglusterfs: fix glusterd statedump crash) posted (#5) for review on master by Atin Mukherjee (amukherj@redhat.com)

--- Additional comment from Vijay Bellur on 2016-08-01 07:23:11 EDT ---

REVIEW: http://review.gluster.org/14987 (libglusterfs: fix glusterd statedump crash) posted (#6) for review on master by Atin Mukherjee (amukherj@redhat.com)

--- Additional comment from Vijay Bellur on 2016-08-04 11:44:13 EDT ---

COMMIT: http://review.gluster.org/14987 committed in master by Jeff Darcy (jdarcy@redhat.com) 
------
commit 049c91565dddb622b8902ccfeb36c0d414c609e1
Author: Atin Mukherjee <amukherj@redhat.com>
Date:   Tue May 31 16:44:48 2016 +0530

    libglusterfs: fix glusterd statedump crash
    
    commit 3c04a91 removed setting typeStr to NULL if num_allocs is set to 0, this
    has caused this regression. Code has been put back like earlier and to avoid
    statedump printing all the NULL values check is modified to see skip the records
    if num_allocs is 0 instead of total_allocs
    
    Change-Id: Ib8bcc2fba908e88cf52b641c3f6bcba74f5e667c
    BUG: 1359190
    Signed-off-by: Atin Mukherjee <amukherj@redhat.com>
    Reviewed-on: http://review.gluster.org/14987
    NetBSD-regression: NetBSD Build System <jenkins@build.gluster.org>
    Smoke: Gluster Build System <jenkins@build.gluster.org>
    CentOS-regression: Gluster Build System <jenkins@build.gluster.org>
    Reviewed-by: N Balachandran <nbalacha@redhat.com>
    Reviewed-by: Prashanth Pai <ppai@redhat.com>
    Reviewed-by: Jeff Darcy <jdarcy@redhat.com>

Comment 1 Vijay Bellur 2016-08-05 05:52:07 UTC
REVIEW: http://review.gluster.org/15091 (libglusterfs: fix glusterd statedump crash) posted (#1) for review on release-3.8 by Atin Mukherjee (amukherj@redhat.com)

Comment 2 Atin Mukherjee 2016-08-05 05:53:59 UTC
*** Bug 1359131 has been marked as a duplicate of this bug. ***

Comment 3 Vijay Bellur 2016-08-07 11:53:51 UTC
COMMIT: http://review.gluster.org/15091 committed in release-3.8 by Niels de Vos (ndevos@redhat.com) 
------
commit ffc67d3fc64a016e9900bdb8b5fed5866850a1d8
Author: Atin Mukherjee <amukherj@redhat.com>
Date:   Tue May 31 16:44:48 2016 +0530

    libglusterfs: fix glusterd statedump crash
    
    Backport of http://review.gluster.org/14987
    
    commit 3c04a91 removed setting typeStr to NULL if num_allocs is set to 0, this
    has caused this regression. Code has been put back like earlier and to avoid
    statedump printing all the NULL values check is modified to see skip the records
    if num_allocs is 0 instead of total_allocs
    
    >Reviewed-on: http://review.gluster.org/14987
    >NetBSD-regression: NetBSD Build System <jenkins@build.gluster.org>
    >Smoke: Gluster Build System <jenkins@build.gluster.org>
    >CentOS-regression: Gluster Build System <jenkins@build.gluster.org>
    >Reviewed-by: N Balachandran <nbalacha@redhat.com>
    >Reviewed-by: Prashanth Pai <ppai@redhat.com>
    >Reviewed-by: Jeff Darcy <jdarcy@redhat.com>
    
    Change-Id: Ib8bcc2fba908e88cf52b641c3f6bcba74f5e667c
    BUG: 1364329
    Signed-off-by: Atin Mukherjee <amukherj@redhat.com>
    Reviewed-on: http://review.gluster.org/15091
    NetBSD-regression: NetBSD Build System <jenkins@build.gluster.org>
    Smoke: Gluster Build System <jenkins@build.gluster.org>
    CentOS-regression: Gluster Build System <jenkins@build.gluster.org>
    Reviewed-by: Prashanth Pai <ppai@redhat.com>
    Reviewed-by: Niels de Vos <ndevos@redhat.com>

Comment 4 Niels de Vos 2016-08-12 09:48:35 UTC
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.8.2, please open a new bug report.

glusterfs-3.8.2 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://www.gluster.org/pipermail/announce/2016-August/000058.html
[2] https://www.gluster.org/pipermail/gluster-users/


Note You need to log in before you can comment on or make changes to this bug.