|Summary:||gfs2_fsck doesn't fix an ea problem.|
|Product:||Red Hat Enterprise Linux 5||Reporter:||Josef Bacik <jbacik>|
|Component:||gfs2-utils||Assignee:||Robert Peterson <rpeterso>|
|Status:||CLOSED ERRATA||QA Contact:||Dean Jansa <djansa>|
|Version:||5.1||CC:||kanderso, rkenna, swhiteho|
|Fixed In Version:||RHBA-2007-0579||Doc Type:||Bug Fix|
|Doc Text:||Story Points:||---|
|Last Closed:||2007-11-07 18:04:40 UTC||Type:||---|
|oVirt Team:||---||RHEL 7.3 requirements from Atomic Host:|
Description Josef Bacik 2007-03-13 16:07:59 UTC
i got an assert in bz 232010 and I went to run gfs2_fsck on the filesystem and I got this [root@rh5cluster1 ~]# gfs2_fsck /dev/gfs2vg/gfs2lv1 Initializing fsck Clearing journals (this may take a while).. Journals cleared. Starting pass1 EA leaf block has incorrect type. gfs2_fsck: buffer count underflow for block 1753066 (0x1abfea) Since I can't do anything until this is fixed I'll try and figure out whats wrong with gfs2_fsck.
Comment 1 Robert Peterson 2007-03-13 17:55:31 UTC
Created attachment 149962 [details] patch to fix the problem This was a simple case of the buffer release code getting called twice. I should probably try to walk through the code and determine if there are any others like it.
Comment 2 Josef Bacik 2007-03-13 20:44:43 UTC
ok after i run the fsck, i'm still getting an invalid metadata assertion when doing an rm -rf on the directory (without anything happening on the other node) GFS2: fsid=rhel5cluster:gfs2lv1.0: fatal: invalid metadata block GFS2: fsid=rhel5cluster:gfs2lv1.0: bh = 1753066 (type: exp=10, found=8) GFS2: fsid=rhel5cluster:gfs2lv1.0: function = ea_foreach_i, file = fs/gfs2/eattr.c, line = 80 GFS2: fsid=rhel5cluster:gfs2lv1.0: about to withdraw this file system GFS2: fsid=rhel5cluster:gfs2lv1.0: telling LM to withdraw GFS2: fsid=rhel5cluster:gfs2lv1.0: withdrawn [<f8beeb4b>] gfs2_lm_withdraw+0x82/0x8d [gfs2] [<f8bffe6e>] gfs2_metatype_check_ii+0x5e/0x6b [gfs2] [<f8be7cfe>] ea_foreach_i+0x86/0x120 [gfs2] [<f8be7dfc>] ea_foreach+0x64/0x1ad [gfs2] [<f8be85ff>] ea_dealloc_unstuffed+0x0/0x362 [gfs2] [<f8be7fdc>] gfs2_ea_dealloc+0x60/0x683 [gfs2] [<c04224d7>] __cond_resched+0x16/0x34 [<f8bebd37>] glock_wait_internal+0x1e5/0x1f7 [gfs2] [<f8bebebb>] gfs2_glock_nq+0x172/0x1a6 [gfs2] [<f8bf8776>] gfs2_delete_inode+0xdc/0x190 [gfs2] [<c0490cbb>] inotify_inode_is_dead+0x18/0x6c [<f8bf86de>] gfs2_delete_inode+0x44/0x190 [gfs2] [<f8bf869a>] gfs2_delete_inode+0x0/0x190 [gfs2] [<c047ecbd>] generic_delete_inode+0xa1/0x107 [<c047e4a6>] iput+0x60/0x62 [<c0477108>] do_unlinkat+0xae/0x119 [<c0407975>] do_syscall_trace+0x124/0x16b [<c0404e4c>] syscall_call+0x7/0xb [<c0610000>] rt_mutex_slowunlock+0x89/0x197 ======================= GFS2: fsid=rhel5cluster:gfs2lv1.0: gfs2_delete_inode: -5
Comment 3 Steve Whitehouse 2007-03-14 09:20:31 UTC
The type 10 is an EA block, so its looking for EAs which are unstuffed from the inode. Type 8 is a log header, which is really odd as that shouldn't be seen anywhere on the disk outside of the log itself.
Comment 4 Robert Peterson 2007-03-14 21:42:59 UTC
For some reason, dinode 0x1ac558 has a non-zero di_eattr pointer to block 0x1ac559. However, block 0x1ac559 is not an eattr block. It looks like a journal log entry. The inode's di_flags are == 0, so it doesn't have the GFS2_DIF_EA_INDIRECT attribute, which is odd. There was a second inode, 0x1ac560, in the same shape, pointing to 0x1ac561, which was also a log header and not an ea. I compared gfs2's eattr.c to gfs1's eattr.c. There were lots of differences, but I didn't find any major problems. Most were cosmetic. The fsck code is very close to the original gfs1 version. Josef said he just used cp -r to copy a kernel tree out there, then he did an rm -fR * to delete it all, which is where he had problems. I partially ran an gfs2_fsck on another fs that josef had, with his new copy of the kernel source tree. There were no eattr problems reported there by fsck.
Comment 5 Robert Peterson 2007-03-14 22:16:29 UTC
I verified that there are no more occurrences of double-brelse as promised in comment #1.
Comment 6 Josef Bacik 2007-03-14 22:23:38 UTC
well if you do an rm -rf and then on another node do a du you get a withdrawl, and then the next time around you have this fs corruption.
Comment 7 Robert Peterson 2007-03-16 18:24:30 UTC
Created attachment 150267 [details] patch to fix the problem [try 2] Josef had several problems: (1) There was the original assert from gfs2, which is out of the scope of this bugzilla record. (2) As stated in comment #4, Josef's file system had some disk inodes that had a di_eattr pointer to a block that wasn't an eattr. That caused gfs2_fsck to enter a little-used code path that had a double-free, which caused fsck to abort abnormally, as noted in comment #1. The original patch fixed that problem, allowing the code to get through the file system completely, but as the description states, (3) gfs2_fsck still didn't fix the corruption. With the previous patch in place, when the bad blocks were encountered, the code was setting the block type to "invalid" then sending back a return code of 1. Later, it was zeroing the di_eattr block number only if the return code < 0. Later still, in pass1c, it was running through all the extended attributes, trying to fix what it could. However, this block wasn't getting processed by pass1c because it was no longer marked as an eattr block (by that time, it was an "invalid" block). Therefore, the di_eattr pointer was never fixed, and therefore, the corruption remained. This version of the patch returns a -1 return code instead. That causes the di_eattr block to be set to 0, which is about all we can do in this case. I ran this on Josef's file system and it correctly set the di_eattr pointer to zero. A subsequent fsck had no issues with the inode then. I have two additional concerns: (1) gfs v1's gfs_fsck has this same problem. Since no one has reported it until gfs2, I have to believe this is a rare kind of corruption for gfs1. So do we crosswrite the fix? If so, where? RHEL4.6? RHEL5.1? It's an easy fix, but we'd have to open another bugzilla record to do it. (2) When the EA damage is fixed, gfs2_fsck doesn't ask the user for permission to repair; it just goes ahead and does it. For most kinds of corruption, fsck doesn't mess with your file system unless it first asks nicely for permission. However, I'm seeing a bunch of cases now where it doesn't ask. It would be easy enough to add a user query for permission. But what is the current user's expectations in this regard? I always thought that all the fsck's would leave your file system alone unless you answered a yes/no question (unless overridden by -y or -n). So what's the "right" thing to do here?
Comment 8 Robert Peterson 2007-03-26 19:17:42 UTC
Fix committed to HEAD and RHEL5 branches in cvs. Setting status to modified.
Comment 11 errata-xmlrpc 2007-11-07 18:04:40 UTC
An advisory has been issued which should help the problem described in this bug report. This report is therefore being closed with a resolution of ERRATA. For more information on the solution and/or where to find the updated files, please follow the link below. You may reopen this bug report if the solution does not work for you. http://rhn.redhat.com/errata/RHBA-2007-0579.html