Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 1510401 - fstat returns ENOENT/ESTALE
Summary: fstat returns ENOENT/ESTALE
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: GlusterFS
Classification: Community
Component: fuse
Version: mainline
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Raghavendra G
QA Contact:
URL:
Whiteboard:
Depends On:
Blocks: 1492591 1529084 1529085 1529086
TreeView+ depends on / blocked
 
Reported: 2017-11-07 10:30 UTC by Raghavendra G
Modified: 2018-03-15 11:20 UTC (History)
1 user (show)

Fixed In Version: glusterfs-4.0.0
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1529084 1529085 1529086 (view as bug list)
Environment:
Last Closed: 2018-03-15 11:20:03 UTC
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:


Attachments (Terms of Use)

Description Raghavendra G 2017-11-07 10:30:38 UTC
Description of problem:
In a multithreaded application if following test is repeated one might end up a failed fstat with ENOENT/ESTALE error:

* Turn off performance.open-behind.
* Thread t1 opens an fd on file 
* file is unlinked or overwritten by a rename - say rename (newfile, file)
* Thread t2 does fstat (fd)

Version-Release number of selected component (if applicable):
Day 0 bug, present in all versions.

How reproducible:
Race but reproducible fairly consistently

Steps to Reproduce:
1. Turn off performance.open-behind
2. Thread t1 opens an fd on file
3. file is unlinked or overwritten by a rename - say rename (newfile, file)
4. Thread t2 does fstat (fd)

Actual results:
sometimes fstat fails with ENOENT/ESTALE

Expected results:
fstat should never fail as there is an fd opened on the file.

Additional info:

Comment 1 Raghavendra G 2017-11-07 10:33:28 UTC
[raghu@unused tmp]$ grep GETATTR estale-ops.txt 
GETATTR {Len:56 Opcode:3 Unique:9895 Nodeid:140295787370744 Uid:601 Gid:601 Pid:18382 Padding:0} {GetattrFlags:0 Dummy:0 Fh:0} 


[raghu@unused tmp]$ grep Pid:18382 ./fuse_30.txt | grep Nodeid:140295787370744
GETATTR {Len:56 Opcode:3 Unique:9895 Nodeid:140295787370744 Uid:601 Gid:601 Pid:18382 Padding:0} {GetattrFlags:0 Dummy:0 Fh:0} 
FLUSH {Len:64 Opcode:25 Unique:9908 Nodeid:140295787370744 Uid:601 Gid:601 Pid:18382 Padding:0} {Fh:140296113765492 Unused:0 Padding:0 LockOwner:15059145825253368282} 
GETATTR {Len:56 Opcode:3 Unique:9910 Nodeid:140295787370744 Uid:601 Gid:601 Pid:18382 Padding:0} {GetattrFlags:0 Dummy:0 Fh:0} 
FLUSH {Len:64 Opcode:25 Unique:9917 Nodeid:140295787370744 Uid:601 Gid:601 Pid:18382 Padding:0} {Fh:140296113764232 Unused:0 Padding:0 LockOwner:15059145825253368282} 
OPEN {Len:48 Opcode:14 Unique:13150 Nodeid:140295787370744 Uid:601 Gid:601 Pid:18382 Padding:0} {Flags:32768 Unused:0} 
FLUSH {Len:64 Opcode:25 Unique:13156 Nodeid:140295787370744 Uid:601 Gid:601 Pid:18382 Padding:0} {Fh:140296113765912 Unused:0 Padding:0 LockOwner:15059145825253368282} 
OPEN {Len:48 Opcode:14 Unique:13193 Nodeid:140295787370744 Uid:601 Gid:601 Pid:18382 Padding:0} {Flags:32768 Unused:0} 
FLUSH {Len:64 Opcode:25 Unique:13202 Nodeid:140295787370744 Uid:601 Gid:601 Pid:18382 Padding:0} {Fh:140296113762832 Unused:0 Padding:0 LockOwner:15059145825253368282}

As we can see above before GETATTR failure (Unique:9895), an open was never done on the same file (Nodeid:140295787370744) by the same application thread (Pid:18382). Since no fd was opened by the same thread, glusterfs falls back to STAT instead of FSTAT which can fail with ESTALE due to rename overwriting the file. However, I do see that fds are opened by other threads on the same file at the time of GETATTR (Unique:9895). So, I am wondering whether the application is written in such a way that fds are opened and consumed across different threads.  If yes, this could the cause for fstat failures as Glusterfs uses an fd only if it is opened by same thread during GETATTR. Otherwise, it'll use STAT instead of FSTAT.

Comment 2 Worker Ant 2017-11-07 10:59:44 UTC
REVIEW: https://review.gluster.org/18681 (mount/fuse: use fstat in getattr implementation if any opened fd is available) posted (#1) for review on master by Raghavendra G

Comment 3 Worker Ant 2017-11-09 17:26:57 UTC
COMMIT: https://review.gluster.org/18681 committed in master by  

------------- mount/fuse: use fstat in getattr implementation if any opened fd is available

The restriction of using fds opened by the same Pid means fds cannot
be shared across threads of multithreaded application. Note that fops
from kernel have different Pid for different threads. Imagine
following sequence of operations:

* Turn off performance.open-behind
* Thread t1 opens an fd - fd1 - on file "file". Let's assume nodeid of
  "file" is "nodeid-file".
* Thread t2 does RENAME ("newfile", "file"). Let's assume nodeid of
  "newfile" as "nodeid-newfile".
* t2 proceeds to do fstat (fd1)

The above set of operations can sometimes result in ESTALE/ENOENT
errors. RENAME overwrites "file" with "newfile" changing its nodeid
from "nodeid-file" to "nodeid-newfile" and post RENAME, "nodeid-file" is
removed from the backend. If fstat carries nodeid-file as argument,
which can happen if lookup has not refreshed the nodeid of "file" and
since t2 doesn't have an fd opened, fuse_getattr_resume uses STAT
which will fail as "nodeid-file" no longer exists.

Since the above set of operations and sharing of fds across
multiple threads are valid, this is a bug.

The fix is to use any fd opened on the inode. In this specific example
fuse_getattr_resume will find fd1 and winds down the call as fstat
(fd1) which won't fail.

Cross-checked with "Miklos Szeredi" <mszeredi.at.redhat.dot.com> for
any security issues with this solution and he approves the solution.

Thanks to "Miklos Szeredi" <mszeredi.at.redhat.dot.com> for all the
pointers and discussions.

Change-Id: I88dd29b3607cd2594eee9d72a1637b5346c8d49c
BUG: 1510401
Signed-off-by: Raghavendra G <rgowdapp@redhat.com>

Comment 4 Worker Ant 2017-12-26 10:13:53 UTC
REVISION POSTED: https://review.gluster.org/19085 (mount/fuse: use fstat in getattr implementation if any opened fd is available) posted (#2) for review on release-3.10 by Raghavendra G

Comment 5 Shyamsundar 2018-03-15 11:20:03 UTC
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-4.0.0, please open a new bug report.

glusterfs-4.0.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://lists.gluster.org/pipermail/announce/2018-March/000092.html
[2] https://www.gluster.org/pipermail/gluster-users/


Note You need to log in before you can comment on or make changes to this bug.