Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 1603056 - When reserve limits are reached, append on an existing file after truncate operation results to hang
Summary: When reserve limits are reached, append on an existing file after truncate op...
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: GlusterFS
Classification: Community
Component: replicate
Version: 4.1
Hardware: Unspecified
OS: Unspecified
unspecified
medium
Target Milestone: ---
Assignee: Ravishankar N
QA Contact:
URL:
Whiteboard:
Depends On: 1599998 1602236 1602241
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-07-19 04:22 UTC by Ravishankar N
Modified: 2018-07-30 19:02 UTC (History)
4 users (show)

Fixed In Version: glusterfs-4.1.2
Doc Type: If docs needed, set a value
Doc Text:
Clone Of: 1602236
Environment:
Last Closed: 2018-07-30 19:02:48 UTC
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:


Attachments (Terms of Use)

Description Ravishankar N 2018-07-19 04:22:33 UTC
+++ This bug was initially created as a clone of Bug #1602236 +++

+++ This bug was initially created as a clone of Bug #1599998 +++

Description of problem:
=======================
When reserve limits are reached, append on an existing file after truncate operation results to hang.

Version-Release number of selected component (if applicable):
3.12.2-13.el7rhgs.x86_64

How reproducible:
always

Steps to Reproduce:
===================
1) Create a Distributed-Replicate volume and start it.
2) Set the storage.reserve limits to 50 using below command,
 gluster v set distrep storage.reserve 50
3) FUSE mount the volume on a client.
4) Write data till the limits reaches reserve limits(you will see 100% mount point fill in df -h client output). I have used fallocate to quickly fill the disk.
5) Pick an existing file and truncate that file.
truncate -s 0 fallocate_99
It will throw ENOSPC error.
6) Now on the same file try to append some data.
cat /etc/redhat-release > fallocate_99

Actual results:
===============
append will hang. cat command hung here.

Expected results:
=================
No hungs.

--- Additional comment from Worker Ant on 2018-07-17 23:57:10 EDT ---

REVIEW: https://review.gluster.org/20527 (afr: switch lk_owner only when pre-op succeeds) posted (#1) for review on master by Ravishankar N

--- Additional comment from Worker Ant on 2018-07-18 12:50:31 EDT ---

COMMIT: https://review.gluster.org/20527 committed in master by "Ravishankar N" <ravishankar@redhat.com> with a commit message- afr: switch lk_owner only when pre-op succeeds

Problem:
In a disk full scenario, we take a failure path in afr_transaction_perform_fop()
and go to unlock phase. But we change the lk-owner before that, causing unlock
to fail. When mount issues another fop that takes locks on that file, it hangs.

Fix:
Change lk-owner only when we are about to perform the fop phase.
Also fix the same issue for arbiters when afr_txn_arbitrate_fop() fails the fop.

Also removed the DISK_SPACE_CHECK_AND_GOTO in posix_xattrop. Otherwise truncate
to zero will fail pre-op phase with ENOSPC when the user is actually trying to
freee up space.

Change-Id: Ic4c8a596b4cdf4a7fc189bf00b561113cf114353
fixes: bz#1602236
Signed-off-by: Ravishankar N <ravishankar@redhat.com>

Comment 1 Ravishankar N 2018-07-19 04:25:03 UTC
https://review.gluster.org/#/c/20532/

Comment 2 Worker Ant 2018-07-19 04:26:56 UTC
REVIEW: https://review.gluster.org/20532 (afr: switch lk_owner only when pre-op succeeds) posted (#1) for review on release-4.1 by Ravishankar N

Comment 3 Worker Ant 2018-07-23 23:42:40 UTC
COMMIT: https://review.gluster.org/20532 committed in release-4.1 by "Shyamsundar Ranganathan" <srangana@redhat.com> with a commit message- afr: switch lk_owner only when pre-op succeeds

Problem:
In a disk full scenario, we take a failure path in afr_transaction_perform_fop()
and go to unlock phase. But we change the lk-owner before that, causing unlock
to fail. When mount issues another fop that takes locks on that file, it hangs.

Fix:
Change lk-owner only when we are about to perform the fop phase.
Also fix the same issue for arbiters when afr_txn_arbitrate_fop() fails the fop.

Also removed the DISK_SPACE_CHECK_AND_GOTO in posix_xattrop. Otherwise truncate
to zero will fail pre-op phase with ENOSPC when the user is actually trying to
freee up space.

Change-Id: Ic4c8a596b4cdf4a7fc189bf00b561113cf114353
fixes: bz#1603056
Signed-off-by: Ravishankar N <ravishankar@redhat.com>
(cherry picked from commit ec0d7d77de3e4bd485a4fa2e53c9137e25c71ce7)

Comment 4 Shyamsundar 2018-07-30 19:02:48 UTC
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-4.1.2, please open a new bug report.

glusterfs-4.1.2 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] https://lists.gluster.org/pipermail/announce/2018-July/000106.html
[2] https://www.gluster.org/pipermail/gluster-users/


Note You need to log in before you can comment on or make changes to this bug.