Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.

Bug 1063781

Summary: NFS ganesha daemon panics if incase the exported gluster volume contains file/directories created directly on the bricks
Product: Red Hat Gluster Storage Reporter: Soumya Koduri <skoduri>
Component: nfs-ganeshaAssignee: Soumya Koduri <skoduri>
Status: CLOSED CURRENTRELEASE QA Contact: storage-qa-internal <storage-qa-internal>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 2.1CC: jthottan, nlevinki, saujain
Target Milestone: ---Flags: jthottan: needinfo? (saujain)
Target Release: ---   
Hardware: Unspecified   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2015-11-27 09:04:48 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Description Soumya Koduri 2014-02-11 12:27:03 UTC
Description of problem:
If incase any gluster volume contains files/directories created on the bricks directly without using any file/mount protocols and if that volume is exported via Ganesha, Ganesha panics when tried to do any I/Os on that volume.

[root@kvm1 ~]# mount -t glusterfs 192.168.122.202:/vol_tmp /tmp/mnt2
[root@kvm1 ~]# 
[root@kvm1 ~]# ls -al /tmp/mnt2
total 16
drwxr-xr-x  5 root root 4096 Feb 11 11:33 .
drwxrwxrwt. 7 root root 4096 Feb 11 11:37 ..
drwxr-xr-x  2 root root 4096 Feb 11 11:26 brick_dir
drwxr-xr-x  2 root root 4096 Feb 11 11:30 brick_dir2
-rw-r--r--  0 root root    0 Feb 11 11:25 brick_file
-rw-r--r--  1 root root    0 Feb 11 11:29 brick_file2
-rw-r--r--  1 root root    0 Feb 11 11:33 brick_file3
-rw-r--r--  1 root root    0 Feb 11 11:15 fuse_mount_file
-rw-r--r--  1 root root    0 Feb 11 11:18 fuse_mount_file2
-rw-r--r--  1 root root    0 Feb 11 11:18 ganesha_file
-rw-r--r--  1 root root    0 Feb 11 11:15 gluster_nfs_file
[root@kvm1 ~]#

"brick_*" are the files created on the brick directories.

[root@kvm1 ~]# /usr/local/bin/ganesha.nfsd -f /home/skoduri/ganesha_basic.conf -L ./nfs-ganesha.log -N NIV_FULL_DEBUG -d
[root@kvm1 ~]# 
[root@kvm1 ~]# showmount -e localhostExport list for localhost:
/vol_tmp *,*
[root@kvm1 ~]# 
[root@kvm1 ~]# mount -t nfs -o vers=3 192.168.122.202:/vol_tmp /tmp/mnt
[root@kvm1 ~]# 
[root@kvm1 ~]# ls -al /tmp/mnt2
total 16
drwxr-xr-x  5 root root 4096 Feb 11 11:33 .
drwxrwxrwt. 7 root root 4096 Feb 11 11:37 ..
drwxr-xr-x  2 root root 4096 Feb 11 11:26 brick_dir
drwxr-xr-x  2 root root 4096 Feb 11 11:30 brick_dir2
-rw-r--r--  0 root root    0 Feb 11 11:25 brick_file
-rw-r--r--  1 root root    0 Feb 11 11:29 brick_file2
-rw-r--r--  1 root root    0 Feb 11 11:33 brick_file3
-rw-r--r--  1 root root    0 Feb 11 11:15 fuse_mount_file
-rw-r--r--  1 root root    0 Feb 11 11:18 fuse_mount_file2
-rw-r--r--  1 root root    0 Feb 11 11:18 ganesha_file
-rw-r--r--  1 root root    0 Feb 11 11:15 gluster_nfs_file
[root@kvm1 ~]# ls -al /tmp/mnt


[root@kvm1 ~]# ps aux | grep ganesha
root      4586  0.0  0.0 103244   804 pts/1    S+   11:37   0:00 grep ganesha
[root@kvm1 ~]# ps aux | grep ls
root      1860  0.0  2.0 585696 20696 ?        Ssl  10:39   0:00 /usr/local/sbin/glusterfsd -s 192.168.122.202 --volfile-id vol_tmp.192.168.122.202.tmp-brick1 -p /var/lib/glusterd/vols/vol_tmp/run/192.168.122.202-tmp-brick1.pid -S /var/run/328194a6f7f429197ec2e8a945992bf4.socket --brick-name /tmp/brick1 -l /usr/local/var/log/glusterfs/bricks/tmp-brick1.log --xlator-option *-posix.glusterd-uuid=9aa351d3-a7ed-4e67-a447-abed0605014f --brick-port 49165 --xlator-option vol_tmp-server.listen-port=49165
root      4584  0.0  0.1 117824  1116 pts/0    D+   11:37   0:00 ls --color=auto -al /tmp/mnt
root      4594  0.0  0.0 103248   808 pts/1    S+   11:37   0:00 grep ls
[root@kvm1 ~]# kill -9 4584



Have verified the same behavior with Gluster-NFS. It lists all the files 
(Note- there is some discrepancy in gluster-nfs as well. If there are more than one file/dir created on the bricks, after a while it prints "Invalid argument" error on the last few files/dirs created. We suspect it might be because of any indexing issue). However it doesn't panic atleast.


[root@kvm1 ~]# ls -al /tmp/mnt
ls: cannot access /tmp/mnt/brick_file3: Invalid argument
total 16
drwxr-xr-x  5 root root 4096 Feb 11 11:33 .
drwxrwxrwt. 7 root root 4096 Feb 11 11:28 ..
drwxr-xr-x  2 root root 4096 Feb 11 11:26 brick_dir
drwxr-xr-x  2 root root 4096 Feb 11 11:30 brick_dir2
-rw-r--r--  0 root root    0 Feb 11 11:25 brick_file
-rw-r--r--  1 root root    0 Feb 11 11:29 brick_file2
-?????????? ? ?    ?       ?            ? brick_file3
-rw-r--r--  1 root root    0 Feb 11 11:15 fuse_mount_file
-rw-r--r--  1 root root    0 Feb 11 11:18 fuse_mount_file2
-rw-r--r--  1 root root    0 Feb 11 11:18 ganesha_file
-rw-r--r--  1 root root    0 Feb 11 11:15 gluster_nfs_file
[root@kvm1 ~]# ls -al /tmp/mnt
ls: cannot access /tmp/mnt/brick_dir2: Invalid argument
ls: cannot access /tmp/mnt/brick_file3: Invalid argument
total 12
drwxr-xr-x  5 root root 4096 Feb 11 11:33 .
drwxrwxrwt. 7 root root 4096 Feb 11 11:28 ..
drwxr-xr-x  2 root root 4096 Feb 11 11:26 brick_dir
d?????????? ? ?    ?       ?            ? brick_dir2
-rw-r--r--  0 root root    0 Feb 11 11:25 brick_file
-rw-r--r--  1 root root    0 Feb 11 11:29 brick_file2
-?????????? ? ?    ?       ?            ? brick_file3
-rw-r--r--  1 root root    0 Feb 11 11:15 fuse_mount_file
-rw-r--r--  1 root root    0 Feb 11 11:18 fuse_mount_file2
-rw-r--r--  1 root root    0 Feb 11 11:18 ganesha_file
-rw-r--r--  1 root root    0 Feb 11 11:15 gluster_nfs_file
[root@kvm1 ~]# 


Version-Release number of selected component (if applicable):
V2.1-dev_5

How reproducible:
Everytime

Steps to Reproduce:
1. Create any file/dir on the bricks manually without using any mount protocols
2. Now start ganesha service and mount the volume containing those files.
3. Run any I/O (lookup) on that mount directory. 

Actual results:
The command 'ls' hangs and the ganesha service dumps the core.

Expected results:
'ls' should list all the files of that volume  (irrespective of how they are created). Atleast ganesha daemon shouldn't panic.

Additional info:
Have to root cause the issue. Will try to address it for Denali release.

Comment 2 Soumya Koduri 2014-07-07 11:10:25 UTC
The panic refers to ganesha daemon crash and its no longer seen in RHS3.0 version of ganesha. 

But the files created on the bricks are still not visible via ganesha. The fix for this is present in upstream and under review. This will be pulled in for the next release. 

This is a minor bug(corner case) once the crash is no longer seen. Its mainly an offline file create case which is not important for tech-preview.

Comment 3 Jiffin 2015-07-16 08:58:49 UTC
This issue is not found in RHGS 3.1

Comment 4 Jiffin 2015-11-27 09:04:48 UTC
This is not produced in the latest ganesha version