Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 1064007 - [RHEV-RHS] gluster fuse mount remains read-only filesystem, after disabling client-side quorum after it is not met
Summary: [RHEV-RHS] gluster fuse mount remains read-only filesystem, after disabling c...
Alias: None
Product: Red Hat Gluster Storage
Classification: Red Hat
Component: glusterfs
Version: 2.1
Hardware: x86_64
OS: Linux
Target Milestone: ---
: ---
Assignee: Bug Updates Notification Mailing List
Depends On:
TreeView+ depends on / blocked
Reported: 2014-02-11 19:09 UTC by SATHEESARAN
Modified: 2015-12-03 17:20 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
virt rhev integration
Last Closed: 2015-12-03 17:20:13 UTC
Target Upstream Version:

Attachments (Terms of Use)
sosreport from RHSS Node1 (deleted)
2014-02-12 05:24 UTC, SATHEESARAN
no flags Details
sosreport from RHSS Node2 (deleted)
2014-02-12 05:25 UTC, SATHEESARAN
no flags Details
Fuse mount log files from Hypervisor (deleted)
2014-02-12 06:30 UTC, SATHEESARAN
no flags Details

Description SATHEESARAN 2014-02-11 19:09:06 UTC
Description of problem:
Following is expected when client-side quorum is enabled, with volume with replica count of 2 -- "First brick of the replica group should always be UP"

When the first brick of the replica group goes down, then that replica group becomes read-only, eventhough the other brick was UP.

Once we hit this state, disabling client-side quorum, doesn't restores the volumes as read/write. The volume remains a read-only filesystem 

Version-Release number of selected component (if applicable):

How reproducible:

Steps to Reproduce:
0. Create a Trusted Storage Pool with 3 RHSS Nodes

1. Create a volume distributed-replicate or replicate with replica count 2

2. Optimize the volume for virt store.
Note: Client-side quorum is enabled by default in virt profile

3. Create a Data Domain in RHEV, with the above created volume

4. Power-off the first node, serving the first bricks of the replica group

5. The fuse mount now becomes read-only

6. Disable client-side quorum on the volume
(i.e) gluster volume reset <vol-name> quorum-type

Actual results:
The fuse mounted volumes still remains a read-only filesystem

Expected results:
As client-side quorum is disabled, the volume should restore to its default (read/write )

Additional info:

Comment 2 Gowrishankar Rajaiyan 2014-02-12 04:21:13 UTC
This looks like a native client-side quorum issue.

Comment 3 SATHEESARAN 2014-02-12 04:55:13 UTC
Here is one more variation of the test case, that I performed :

With the same setup as in comment0 :

1. Created a trusted storage pool of 3 RHSS Nodes ( NODE1, NODE2, NODE3 )
2. Created a 2X2 Distributed-replicate volume
    Here bricks are from NODE1 and NODE2 only
   (i.e) gluster volume create <vol-name> replica 2 NODE1:<BRICK1> NODE2:<BRICK1> NODE1:<BRICK2> NODE2:<BRICK2>

3. Optimized the volume for virt-store
4. Started the volume
5. Created a Data domain with this gluster volume
6. Now, killed only the first brick of the replica group
   Note: get the pid of brick process using 'gluster volume status' command
   (i.e) pkill -9 <pid>
7. Now the volume becomes read-only
8. Disabled client-side quorum
(i.e) gluster volume reset <vol-name> quorum-type
9. Now the volume has restored itself with read-write

What I understand from this test is this :

1. When the first brick of all replica groups goes down, volume goes read-only and then disabling client-side quorum, never brings the volume back to read-write mode

2. When first brick of one of the replica group goes down, that replica group goes read-only, and then disabling client-side quorum, bring the replica group back to read-write mode

Comment 4 SATHEESARAN 2014-02-12 05:24:06 UTC
Created attachment 862119 [details]
sosreport from RHSS Node1

sosreports from RHSS Node1

Comment 5 SATHEESARAN 2014-02-12 05:25:59 UTC
Created attachment 862120 [details]
sosreport from RHSS Node2

sosreports from RHSS Node2,

Comment 6 SATHEESARAN 2014-02-12 06:30:30 UTC
Created attachment 862123 [details]
Fuse mount log files from Hypervisor

This is the fuse mount log from Hypervisor

Comment 7 SATHEESARAN 2014-02-12 08:31:33 UTC
As Pranith suggested, I repeated the test in comment0 with write-behind disabled

Turned off write-behind ( all perf xlators are by default, turned off in virt profile )

Here is the test,
0. Turn off the write-behind on the volume
(i.e) gluster volume set <vol-name> write-behind off

1. Killed all the first bricks on the volume
2. Volume has become read-only
3. Disabled client-side quorum
(i.e) gluster volume reset <vol-name> quorum-type none
4. Now the volume has become read-write. 
NOTE: This doesn't happened when write-behind is on

Comment 8 SATHEESARAN 2014-02-12 08:58:33 UTC
(In reply to SATHEESARAN from comment #7)

> 3. Disabled client-side quorum
> (i.e) gluster volume reset <vol-name> quorum-type none

Sorry for the typo
disabling quorum can be done using :
gluster volume reset <vol-name> quorum-type

Comment 9 Vivek Agarwal 2015-12-03 17:20:13 UTC
Thank you for submitting this issue for consideration in Red Hat Gluster Storage. The release for which you requested us to review, is now End of Life. Please See

If you can reproduce this bug against a currently maintained version of Red Hat Gluster Storage, please feel free to file a new report against the current release.

Note You need to log in before you can comment on or make changes to this bug.