Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 1596904 - remove-brick is failing to remove hidden dot files
Summary: remove-brick is failing to remove hidden dot files
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: GlusterFS
Classification: Community
Component: object-storage
Version: mainline
Hardware: x86_64
OS: Linux
unspecified
unspecified
Target Milestone: ---
Assignee: bugs@gluster.org
QA Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-06-30 03:50 UTC by Chris King
Modified: 2018-07-24 04:25 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-07-24 04:25:43 UTC
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:


Attachments (Terms of Use)

Description Chris King 2018-06-30 03:50:33 UTC
Description of problem:

after completing "remove-brick", the removed drive is still full of the hidden "dot files" (eg .htaccess). However, the dot files where still relocated to new bricks on the volume (not lost).


Version-Release number of selected component (if applicable):

glusterfs 4.0.2


How reproducible:

I have only run this once, but both bricks still contained the all of the hidden dot files.


Steps to Reproduce:
1. ensure that you have hidden files on volume
2. gluster volume remove-brick conoak-photos liberator:/mnt/disk1/conoak-photos-data liberator:/mnt/disk3/conoak-photos-data start
3. gluster volume remove-brick conoak-photos liberator:/mnt/disk1/conoak-photos-data liberator:/mnt/disk3/conoak-photos-data status
4. wait until status is "completed"
5. gluster volume remove-brick conoak-photos liberator:/mnt/disk1/conoak-photos-data liberator:/mnt/disk3/conoak-photos-data commit
6. find /mnt/disk1 -type f | grep -v "/[.]glusterfs/"
7. find /mnt/disk3 -type f | grep -v "/[.]glusterfs/"


Actual results:

hundreds of files still be found on the removed brick
eg
/mnt/disk1/conoak-photos-data/Photos/2003/2003-05-02/downsize/.picasa.ini
/mnt/disk1/conoak-photos-data/Photos/2003/2003-05-02/.picasa.ini
/mnt/disk1/conoak-photos-data/Photos/2003/2003-05-25/.picasa.ini
/mnt/disk1/conoak-photos-data/Photos/2003/2003-05-30/.picasa.ini


Expected results:

Expected no files on the removed brick.


Additional info:

one of the bricks has zero sized files and had the "T" attribute, the other brick and volume where original size;

ls -la /mnt/disk1/conoak-photos-data/Photos/2003/2003-05-02/.picasa.ini
---------T 2 chris nasusers 0 Mar  7 07:10 /mnt/disk1/conoak-photos-data/Photos/2003/2003-05-02/.picasa.ini

ls -la /mnt/disk5/conoak-photos-data/Photos/2003/2003-05-02/.picasa.ini
-rw-rw-r-- 2 chris nasusers 323 Jan 24  2013 /mnt/disk5/conoak-photos-data/Photos/2003/2003-05-02/.picasa.ini

ls -la /mnt/gluster-conoak-photos/Photos/2003/2003-05-02/.picasa.ini
-rw-rw-r-- 1 chris nasusers 323 Jan 24  2013 /mnt/gluster-conoak-photos/Photos/2003/2003-05-02/.picasa.ini

Comment 1 Nithya Balachandran 2018-07-16 09:41:59 UTC
(In reply to Chris King from comment #0)
> Description of problem:
> 
> after completing "remove-brick", the removed drive is still full of the
> hidden "dot files" (eg .htaccess). However, the dot files where still
> relocated to new bricks on the volume (not lost).
> 
> 
> Version-Release number of selected component (if applicable):
> 
> glusterfs 4.0.2
> 
> 
> How reproducible:
> 
> I have only run this once, but both bricks still contained the all of the
> hidden dot files.
> 
> 
> Steps to Reproduce:
> 1. ensure that you have hidden files on volume
> 2. gluster volume remove-brick conoak-photos
> liberator:/mnt/disk1/conoak-photos-data
> liberator:/mnt/disk3/conoak-photos-data start
> 3. gluster volume remove-brick conoak-photos
> liberator:/mnt/disk1/conoak-photos-data
> liberator:/mnt/disk3/conoak-photos-data status
> 4. wait until status is "completed"
> 5. gluster volume remove-brick conoak-photos
> liberator:/mnt/disk1/conoak-photos-data
> liberator:/mnt/disk3/conoak-photos-data commit
> 6. find /mnt/disk1 -type f | grep -v "/[.]glusterfs/"
> 7. find /mnt/disk3 -type f | grep -v "/[.]glusterfs/"
> 
> 
> Actual results:
> 
> hundreds of files still be found on the removed brick
> eg
> /mnt/disk1/conoak-photos-data/Photos/2003/2003-05-02/downsize/.picasa.ini
> /mnt/disk1/conoak-photos-data/Photos/2003/2003-05-02/.picasa.ini
> /mnt/disk1/conoak-photos-data/Photos/2003/2003-05-25/.picasa.ini
> /mnt/disk1/conoak-photos-data/Photos/2003/2003-05-30/.picasa.ini
> 
> 
> Expected results:
> 
> Expected no files on the removed brick.
> 
> 
> Additional info:
> 
> one of the bricks has zero sized files and had the "T" attribute, the other
> brick and volume where original size;
> 
> ls -la /mnt/disk1/conoak-photos-data/Photos/2003/2003-05-02/.picasa.ini
> ---------T 2 chris nasusers 0 Mar  7 07:10
> /mnt/disk1/conoak-photos-data/Photos/2003/2003-05-02/.picasa.ini
> 
> ls -la /mnt/disk5/conoak-photos-data/Photos/2003/2003-05-02/.picasa.ini
> -rw-rw-r-- 2 chris nasusers 323 Jan 24  2013
> /mnt/disk5/conoak-photos-data/Photos/2003/2003-05-02/.picasa.ini
> 
> ls -la /mnt/gluster-conoak-photos/Photos/2003/2003-05-02/.picasa.ini
> -rw-rw-r-- 1 chris nasusers 323 Jan 24  2013
> /mnt/gluster-conoak-photos/Photos/2003/2003-05-02/.picasa.ini


Are all the files on the removed bricks zero sized 'T' files? If yes, these are internal files and can be ignored.

Comment 2 Chris King 2018-07-19 06:10:24 UTC
> Are all the files on the removed bricks zero sized 'T' files? If yes, these
> are internal files and can be ignored.

just on one brick, the second brick still had the original files

Comment 3 Nithya Balachandran 2018-07-23 08:58:15 UTC
(In reply to Chris King from comment #2)
> > Are all the files on the removed bricks zero sized 'T' files? If yes, these
> > are internal files and can be ignored.
> 
> just on one brick, the second brick still had the original files

Hi Chris,

I need more info to figure this out. IIUC,the bricks that were removed were disk1 and disk3. So disk5, where you see the original files, is still a part of your volume.

gluster volume remove-brick conoak-photos liberator:/mnt/disk1/conoak-photos-data liberator:/mnt/disk3/conoak-photos-data start


What is the output of gluster volume info <volname>?
Having zero sized 'T' files on the removed bricks (disk1 and disk3) is not a problem.

Comment 4 Chris King 2018-07-23 22:42:58 UTC
(In reply to Nithya Balachandran from comment #3)
> Hi Chris,
> 
> I need more info to figure this out. IIUC,the bricks that were removed were
> disk1 and disk3. So disk5, where you see the original files, is still a part
> of your volume.

Nithya, the overall exercise was to add two 8TB SAS drives, expand the volumes over bricks on these drives (which went well) and then remove two two 6TB SATA drives.

I should have shown the listing of disk3 (not disk 5) my mistake. Disk 3 was also zero sized.

# ls -la /mnt/disk3/conoak-photos-data/Photos/2003/2003-05-02/.picasa.ini
---------T 2 chris nasusers 0 Mar  7 07:10 /mnt/disk3/conoak-photos-data/Photos/2003/2003-05-02/.picasa.ini


> What is the output of gluster volume info <volname>?

# gluster volume info conoak-photos-data
Volume conoak-photos-data does not exist
[root@liberator ~]# gluster volume info conoak-photos

Volume Name: conoak-photos
Type: Distributed-Replicate
Volume ID: 9d5c2238-6f37-4f75-a2ea-5348b174fbb1
Status: Started
Snapshot Count: 0
Number of Bricks: 2 x 2 = 4
Transport-type: tcp
Bricks:
Brick1: liberator:/mnt/disk5/conoak-photos-data
Brick2: liberator:/mnt/disk7/conoak-photos-data
Brick3: liberator:/mnt/disk2/conoak-photos-data
Brick4: liberator:/mnt/disk6/conoak-photos-data
Options Reconfigured:
changelog.changelog: on
geo-replication.ignore-pid-check: on
geo-replication.indexing: on
transport.address-family: inet
nfs.disable: on
performance.client-io-threads: off


> Having zero sized 'T' files on the removed bricks (disk1 and disk3) is not a
> problem.

OK its not a serious, but it does not seem right that all files except the "dot files" are removed completely and only the dot files remain (even if they are zero sized)

Comment 5 Nithya Balachandran 2018-07-24 04:25:43 UTC
(In reply to Chris King from comment #4)
> (In reply to Nithya Balachandran from comment #3)
> > Hi Chris,
> > 
> > I need more info to figure this out. IIUC,the bricks that were removed were
> > disk1 and disk3. So disk5, where you see the original files, is still a part
> > of your volume.
> 
> Nithya, the overall exercise was to add two 8TB SAS drives, expand the
> volumes over bricks on these drives (which went well) and then remove two
> two 6TB SATA drives.
> 
> I should have shown the listing of disk3 (not disk 5) my mistake. Disk 3 was
> also zero sized.
> 
> # ls -la /mnt/disk3/conoak-photos-data/Photos/2003/2003-05-02/.picasa.ini
> ---------T 2 chris nasusers 0 Mar  7 07:10
> /mnt/disk3/conoak-photos-data/Photos/2003/2003-05-02/.picasa.ini
> 
> 
> > What is the output of gluster volume info <volname>?
> 
> # gluster volume info conoak-photos-data
> Volume conoak-photos-data does not exist
> [root@liberator ~]# gluster volume info conoak-photos
> 
> Volume Name: conoak-photos
> Type: Distributed-Replicate
> Volume ID: 9d5c2238-6f37-4f75-a2ea-5348b174fbb1
> Status: Started
> Snapshot Count: 0
> Number of Bricks: 2 x 2 = 4
> Transport-type: tcp
> Bricks:
> Brick1: liberator:/mnt/disk5/conoak-photos-data
> Brick2: liberator:/mnt/disk7/conoak-photos-data
> Brick3: liberator:/mnt/disk2/conoak-photos-data
> Brick4: liberator:/mnt/disk6/conoak-photos-data
> Options Reconfigured:
> changelog.changelog: on
> geo-replication.ignore-pid-check: on
> geo-replication.indexing: on
> transport.address-family: inet
> nfs.disable: on
> performance.client-io-threads: off
> 
> 
> > Having zero sized 'T' files on the removed bricks (disk1 and disk3) is not a
> > problem.
> 
> OK its not a serious, but it does not seem right that all files except the
> "dot files" are removed completely and only the dot files remain (even if
> they are zero sized)

It is possible that those linkto files (the 'T' files) were already present on those bricks for some reason (maybe the files were renamed at some point).  Gluster will not migrate linkto files - there is no point doing so as they are internal files.


I'm closing this BZ as this is not a bug.


Note You need to log in before you can comment on or make changes to this bug.