Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 1516313 - Bringing down data bricks in cyclic order results in arbiter brick becoming the source for heal.
Summary: Bringing down data bricks in cyclic order results in arbiter brick becoming t...
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: GlusterFS
Classification: Community
Component: arbiter
Version: 3.13
Hardware: x86_64
OS: Linux
high
unspecified
Target Milestone: ---
Assignee: Karthik U S
QA Contact:
URL:
Whiteboard:
Depends On: 1482064 1566131
Blocks: 1401969 glusterfs-3.13.2
TreeView+ depends on / blocked
 
Reported: 2017-11-22 13:02 UTC by Karthik U S
Modified: 2018-04-11 15:12 UTC (History)
12 users (show)

Fixed In Version: glusterfs-3.13.2
Doc Type: If docs needed, set a value
Doc Text:
Clone Of: 1482064
Environment:
Last Closed: 2018-01-23 21:37:19 UTC
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:


Attachments (Terms of Use)

Comment 1 Worker Ant 2017-11-22 13:05:02 UTC
REVIEW: https://review.gluster.org/18843 (cluster/afr: Fix for arbiter becoming source) posted (#1) for review on release-3.13 by Karthik U S

Comment 2 Worker Ant 2017-11-27 18:13:57 UTC
COMMIT: https://review.gluster.org/18843 committed in release-3.13 by \"Karthik U S\" <ksubrahm@redhat.com> with a commit message- cluster/afr: Fix for arbiter becoming source

Problem:
When eager-lock is on, and two writes happen in parallel on a FD
we were observing the following behaviour:
- First write fails on one data brick
- Since the post-op is not yet happened, the inode refresh will get
  both the data bricks as readable and set it in the inode context
- In flight split brain check see both the data bricks as readable
  and allows the second write
- Second write fails on the other data brick
- Now the post-op happens and marks both the data bricks as bad and
  arbiter will become source for healing

Fix:
Adding one more variable called write_suvol in inode context and it
will have the in memory representation of the writable subvols. Inode
refresh will not update this value and its lifetime is pre-op through
unlock in the afr transaction. Initially the pre-op will set this
value same as read_subvol in inode context and then in the in flight
split brain check we will use this value instead of read_subvol.
After all the checks we will update the value of this and set the
read_subvol same as this to avoid having incorrect value in that.

Change-Id: I2ef6904524ab91af861d59690974bbc529ab1af3
BUG: 1516313
Signed-off-by: karthik-us <ksubrahm@redhat.com>
(cherry picked from commit 19f9bcff4aada589d4321356c2670ed283f02c03)

Comment 3 Shyamsundar 2017-12-08 17:46:04 UTC
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.13.0, please open a new bug report.

glusterfs-3.13.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://lists.gluster.org/pipermail/announce/2017-December/000087.html
[2] https://www.gluster.org/pipermail/gluster-users/

Comment 4 Karthik U S 2018-01-03 06:01:17 UTC
Found some issues in the patch which can again lead to the same problem of arbiter becoming source. Changing the state back to Post.

Comment 5 Worker Ant 2018-01-15 06:39:16 UTC
REVIEW: https://review.gluster.org/19192 (cluster/afr: Fixing the flaws in arbiter becoming source patch) posted (#1) for review on release-3.13 by Karthik U S

Comment 6 Worker Ant 2018-01-18 18:20:31 UTC
COMMIT: https://review.gluster.org/19192 committed in release-3.13 by \"Karthik U S\" <ksubrahm@redhat.com> with a commit message- cluster/afr: Fixing the flaws in arbiter becoming source patch

Problem:
Setting the write_subvol value to read_subvol in case of metadata
transaction during pre-op (commit 19f9bcff4aada589d4321356c2670ed283f02c03)
might lead to the original problem of arbiter becoming source.

Scenario:
1) All bricks are up and good
2) 2 writes w1 and w2 are in progress in parallel
3) ctx->read_subvol is good for all the subvolumes
4) w1 succeeds on brick0 and fails on brick1, yet to do post-op on
   the disk
5) read/lookup comes on the same file and refreshes read_subvols back
   to all good
6) metadata transaction happens which makes ctx->write_subvol to be
   assigned with ctx->read_subvol which is all good
7) w2 succeeds on brick1 and fails on brick0 and this will update the
   brick in reverse order leading to arbiter becoming source

Fix:
Instead of setting the ctx->write_subvol to ctx->read_subvol in the
pre-op statge, if there is a metadata transaction, check in the
function __afr_set_in_flight_sb_status() if it is a data/metadata
transaction. Use the value of ctx->write_subvol if it is a data
transactions and ctx->read_subvol value for other transactions.

With this patch we assign the value of ctx->write_subvol in the
afr_transaction_perform_fop() with the on disk value, instead of
assigning it in the afr_changelog_pre_op() with the in memory value.

Change-Id: Id2025a7e965f0578af35b1abaac793b019c43cc4
BUG: 1516313
Signed-off-by: karthik-us <ksubrahm@redhat.com>
(cherry picked from commit ba149bac92d169ae2256dbc75202dc9e5d06538e)

Comment 7 Shyamsundar 2018-01-23 21:37:19 UTC
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.13.2, please open a new bug report.

glusterfs-3.13.2 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://lists.gluster.org/pipermail/announce/2018-January/000089.html
[2] https://www.gluster.org/pipermail/gluster-users/


Note You need to log in before you can comment on or make changes to this bug.