Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 1060676 - [add-brick]: I/O on NFS fails when bricks are added to a distribute-replicate volume
Summary: [add-brick]: I/O on NFS fails when bricks are added to a distribute-replicate...
Alias: None
Product: Red Hat Gluster Storage
Classification: Red Hat
Component: distribute
Version: rhgs-3.0
Hardware: x86_64
OS: Linux
Target Milestone: ---
: RHGS 3.1.2
Assignee: Nithya Balachandran
QA Contact:
Whiteboard: dht-add-brick
Depends On:
Blocks: 1260783
TreeView+ depends on / blocked
Reported: 2014-02-03 10:40 UTC by Sachidananda Urs
Modified: 2016-03-01 05:22 UTC (History)
10 users (show)

Fixed In Version: glusterfs-3.7.5-6
Doc Type: Bug Fix
Doc Text:
Clone Of:
Last Closed: 2016-03-01 05:22:22 UTC
Target Upstream Version:

Attachments (Terms of Use)
NFS client logs (deleted)
2014-02-03 10:40 UTC, Sachidananda Urs
no flags Details

System ID Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2016:0193 normal SHIPPED_LIVE Red Hat Gluster Storage 3.1 update 2 2016-03-01 10:20:36 UTC

Description Sachidananda Urs 2014-02-03 10:40:18 UTC
Created attachment 858495 [details]
NFS client logs

Description of problem:

When new bricks are added to distribute-replicate volume, IO fails on NFS mount with error messages:

dd: opening `dir.94/file.100': No such file or directory
mkdir: cannot create directory `dir.95': Invalid argument
dd: opening `dir.95/file.1': No such file or directory

Attaching nfs logs. No errors reported in glusterd logs.

Version-Release number of selected component (if applicable):

glusterfs built on Jan 25 2014 07:04:06

How reproducible:

Steps to Reproduce:
1. Create a 2x2 volume, and do some IO on nfs mount
2. Peer probe two more machine
3. add-brick to the cluster

IO will fail on the mount

Expected results:

add-brick should be seamless.

Comment 2 santosh pradhan 2014-06-24 08:28:40 UTC
NFS protocol has no relation with brick operation like adding/removing or rebalance. This needs to be looked by DHT or AFR why the NFS fop fails, mostly DHT team.

Comment 3 Shyamsundar 2014-08-13 13:53:41 UTC
From the logs, I see that there are errors in the dht_access function which had an issue of treating directories as files in certain cases, where the cluster is expanded (i.e bricks added etc.).

This is being fixed as a part of the bug #1125824

Once fixed there and downstream, would like to repeat this test case to ensure that this problem is not present.

Comment 4 Raghavendra G 2015-11-10 05:16:33 UTC
A duplicate of:

Fixed by:

With Patch #61036 and fixes to dht-access, this issue should be fixed.

Comment 6 RajeshReddy 2015-11-23 07:40:32 UTC
Tested with build glusterfs-server-3.7.5-6, created 2x2 volume and mounted it on client using nfs and created 200 deep directories and cd to the leaf directory (../dir199/dir200) and then added two new bricks to the volume and while re-balance is going on, from the client able to run ls  and mkdir so marking this bug a verified

Comment 8 errata-xmlrpc 2016-03-01 05:22:22 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

Note You need to log in before you can comment on or make changes to this bug.