Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 1356396 - [crash kernel panic] Internal error XFS_WANT_CORRUPTED_RETURN
Summary: [crash kernel panic] Internal error XFS_WANT_CORRUPTED_RETURN
Keywords:
Status: CLOSED DUPLICATE
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: kexec-tools
Version: 7.3
Hardware: All
OS: Linux
urgent
urgent
Target Milestone: rc
: ---
Assignee: kdump team
QA Contact: Qiao Zhao
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-07-14 05:16 UTC by Qiao Zhao
Modified: 2016-10-11 08:26 UTC (History)
10 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2016-10-11 08:26:13 UTC


Attachments (Terms of Use)

Description Qiao Zhao 2016-07-14 05:16:53 UTC
Description of problem:

2nd kernel occur panic when i trigger general kdump testing:

[    3.093155] XFS (dm-0): Internal error XFS_WANT_CORRUPTED_RETURN at line 208  
of file fs/xfs/libxfs/xfs_dir2_data.c.  Caller xfs_dir3_block_verify+0xd0/0xf0 [ 
xfs]                                                                             
[    3.093161] CPU: 0 PID: 708 Comm: mount Not tainted 3.10.0-467.el7.s390x #1   
[    3.093163]        000000007193b2b0 000000007193b340 0000000000000002 0000000 
000000000                                                                        
       000000007193b3e0 000000007193b358 000000007193b358 0000000000686e46       
       0000000000000000 000000000082bd1c 00000000008104a8 000003ff0000000b       
       000000007193b3a0 000000007193b340 0000000000000000 0000000000000000       
       000003ff804874e0 00000000001165c8 000000007193b340 000000007193b3a0       
[    3.093173] Call Trace:                                                       
[    3.093174] ([<00000000001164a4>] show_trace+0x104/0x158)                     
[    3.093181]  [<000000000011657c>] show_stack+0x84/0x100                       
[    3.093182]  [<000003ff8041a4b0>] __xfs_dir3_data_check+0x818/0x920 [xfs]     
[    3.093195]  [<000003ff80417c38>] xfs_dir3_block_verify+0xd0/0xf0 [xfs]       
[    3.093208]  [<000003ff80417dc0>] xfs_dir3_block_write_verify+0x58/0x120 [xfs 
]                                                                                
[    3.093221]  [<000003ff8043a2de>] _xfs_buf_ioapply+0x9e/0x468 [xfs]           
[    3.093239]  [<000003ff8043a7b8>] xfs_buf_submit+0x88/0x198 [xfs]             
[    3.093253]  [<000003ff8043aa1e>] __xfs_buf_delwri_submit+0x156/0x258 [xfs]   
[    3.093268]  [<000003ff8043b954>] xfs_buf_delwri_submit+0x5c/0xc8 [xfs]       
[    3.093283]  [<000003ff8047008e>] xlog_recover_commit_trans+0x1fe/0x2e8 [xfs] 
                                                                                 
[    3.093304]  [<000003ff8047027c>] xlog_recovery_process_trans+0x104/0x120 [xf 
s]                                                                               
[    3.093321]  [<000003ff80470346>] xlog_recover_process_ophdr+0xae/0x128 [xfs] 
                                                                                 
[    3.093337]  [<000003ff80470474>] xlog_recover_process_data+0xb4/0x108 [xfs]  
[    3.093354]  [<000003ff804705da>] xlog_recover_process+0x112/0x170 [xfs]      
[    3.093370]  [<000003ff80470aa8>] xlog_do_recovery_pass+0x470/0x5f0 [xfs]     
[    3.093387]  [<000003ff80470cdc>] xlog_do_log_recovery+0xb4/0x110 [xfs]       
[    3.093403]  [<000003ff80470d78>] xlog_do_recover+0x40/0x140 [xfs]            
[    3.093419]  [<000003ff80471766>] xlog_recover+0xde/0x1a8 [xfs]               
[    3.093437]  [<000003ff80461454>] xfs_log_mount+0x114/0x2c8 [xfs]             
[    3.093453]  [<000003ff80455b9c>] xfs_mountfs+0x48c/0x7d0 [xfs]               
[    3.093470]  [<000003ff8045a6c6>] xfs_fs_fill_super+0x496/0x5c0 [xfs]         
[    3.093486]  [<00000000002d6b2a>] mount_bdev+0x1c2/0x1f0                      
[    3.093490]  [<000003ff80457e3a>] xfs_fs_mount+0x42/0x50 [xfs]                
[    3.093506]  [<00000000002d75c8>] mount_fs+0x70/0x1a8                         
[    3.093508]  [<00000000002fb3a6>] vfs_kern_mount+0x7e/0x128                   
[    3.093512]  [<00000000002fe51a>] do_mount+0x25a/0xa40                        
[    3.093514]  [<00000000002fedc0>] SyS_mount+0xc0/0x108                        
[    3.093516]  [<0000000000694154>] sysc_nr_ok+0x22/0x28                        
[    3.093520]  [<000003fffd387522>] 0x3fffd387522                               
[    3.093538] XFS (dm-0): Metadata corruption detected at xfs_dir3_block_write_ 
verify+0x10e/0x120 [xfs], xfs_dir3_block block 0x1985100                         
[    3.093539] XFS (dm-0): Unmount and run xfs_repair                            
[    3.093540] XFS (dm-0): First 64 bytes of corrupted metadata buffer:          
[    3.093543] 0000000070451000: 58 44 42 33 55 fe 3f ba 00 00 00 00 01 98 51 00 
  XDB3U.?.......Q.                                                               
[    3.093544] 0000000070451010: 00 00 00 0b 00 00 2c 18 cc 2c b5 ae ed c2 49 3b 
  ......,..,....I;                                                               
[    3.093545] 0000000070451020: b9 f8 de f1 7e 63 18 ee 00 00 00 00 02 0b d8 ac 
  ....~c..........                                                               
[    3.093547] 0000000070451030: 0a 58 03 10 00 a8 00 18 01 08 00 18 00 00 00 20 
  .X.............                                                                
[    3.093549] XFS (dm-0): xfs_do_force_shutdown(0x8) called from line 1288 of f 
ile fs/xfs/xfs_buf.c.  Return address = 0x000003ff8043a69a                       
[    3.093550] XFS (dm-0): Corruption of in-memory data detected.  Shutting down 
 filesystem                                                                      
[    3.093551] XFS (dm-0): Please umount the filesystem and rectify the problem( 
s)                                                                               
[    3.095754] XFS (dm-0): log mount/recovery failed: error -117                 
[    3.095833] XFS (dm-0): log mount failed                                      
[ [1;31mFAILED [0m] Failed to mount /sysroot.                                    
See 'systemctl status sysroot.mount' for details.                                
[ [1;33mDEPEND [0m] Dependency failed for Initrd Root File System.               
[ [1;33mDEPEND [0m] Dependency failed for Reload Configuration from the Real Roo 
t.                                                                               
[ [32m  OK   [0m] Stopped dracut pre-udev hook.                                  
         Stopping dracut pre-udev hook...                                        
[ [32m  OK   [0m] Stopped dracut cmdline hook.                                   
         Stopping dracut cmdline hook...                                         
[ [32m  OK   [0m] Stopped dracut initqueue hook.                                 
         Stopping dracut initqueue hook...                                       
[ [32m  OK   [0m] Reached target Initrd File Systems.                            
[ [32m  OK   [0m] Stopped target Basic System.                                   
[ [32m  OK   [0m] Stopped target System Initialization.                          
[ [32m  OK   [0m] Started Emergency Shell.                                       
         Starting Emergency Shell...                                             
[ [32m  OK   [0m] Reached target Emergency Mode.                                 
                                                                                 
Generating "/run/initramfs/rdsosreport.txt"      

kernel-3.10.0-467.el7
kexec-tools-2.0.7-42.el7.x86_64


Version-Release number of selected component (if applicable):


How reproducible:
20%

Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:
kdump normal.

Additional info:

Comment 2 Dave Young 2016-07-15 06:49:52 UTC
Interesting, it is the first time we met a corruption.

Dave, any thoughts?

Comment 3 Dave Chinner 2016-07-16 00:14:21 UTC
NOthing to go on here. There's a corruption detected during log recovery, which means something has gone wrong before this filesystem was mounted.

Keep in mind that "kdump testing" means nothing to me - I've got no idea what testing is being performed what platform, the storag ein use, the fielsystem configuration, the context, what filesystem had the error, what happened prior to  triggering kdump etc. There's not much we can do with just a random "something got corrupted" reports like this.

-Dave.

Comment 4 Dave Young 2016-07-18 01:48:26 UTC
Qiao, could you check if you can reproduce it reliability? If so please provide enough detail information as Dave replied in comment #3.

Comment 5 Qiao Zhao 2016-07-18 06:50:13 UTC
(In reply to Dave Young from comment #4)
> Qiao, could you check if you can reproduce it reliability? If so please
> provide enough detail information as Dave replied in comment #3.

Hi Dave, I met this problem again, please look:
job: https://beaker.engineering.redhat.com/recipes/2872991
console log: http://beaker-archive.app.eng.bos.redhat.com/beaker-logs/2016/07/14043/1404381/2872991/console.log

] Started File System Check on /dev/d...ed0-a3ac-4ee7-855d-c84c86cef3a3.  
         Mounting /sysroot...  
[    6.184053] SGI XFS with ACLs, security attributes, no debug enabled 
[    6.194266] XFS (sda9): Mounting V5 Filesystem 
[    6.324227] XFS (sda9): Starting recovery (logdev: internal) 
[    6.441954] XFS (sda9): Internal error XFS_WANT_CORRUPTED_RETURN at line 188 of file fs/xfs/libxfs/xfs_dir2_data.c.  Caller xfs_dir3_block_verify+0x9a/0xb0 [xfs] 
[    6.458086] CPU: 11 PID: 516 Comm: mount Not tainted 3.10.0-469.el7.x86_64 #1 
[    6.466059] Hardware name: IBM System x3650 M4 -[7915IK1]-/00W2609, BIOS -[VVE120EUS-1.21]- 10/11/2012 
[    6.476457]  ffff88086b520cd0 00000000b10406e0 ffff88086c0476f8 ffffffff81684ee9 
[    6.484767]  ffff88086c047710 ffffffffa032391b ffffffffa030399a ffff88086c047798 
[    6.493075]  ffffffffa0305c69 ffff88086b520ff8 ffff88086d7cd580 00000009812e995e 
[    6.501381] Call Trace: 
[    6.504122]  [<ffffffff81684ee9>] dump_stack+0x19/0x1b 
[    6.509885]  [<ffffffffa032391b>] xfs_error_report+0x3b/0x40 [xfs] 
[    6.516803]  [<ffffffffa030399a>] ? xfs_dir3_block_verify+0x9a/0xb0 [xfs] 
[    6.524399]  [<ffffffffa0305c69>] __xfs_dir3_data_check+0x459/0x560 [xfs] 
[    6.531995]  [<ffffffffa030399a>] xfs_dir3_block_verify+0x9a/0xb0 [xfs] 
[    6.539394]  [<ffffffffa0303ba4>] xfs_dir3_block_write_verify+0x44/0x110 [xfs] 
[    6.547466]  [<ffffffffa0321418>] ? __xfs_buf_delwri_submit+0x168/0x250 [xfs] 
[    6.555449]  [<ffffffffa031f7e1>] _xfs_buf_ioapply+0x91/0x460 [xfs] 
[    6.562443]  [<ffffffff810c2c10>] ? wake_up_state+0x20/0x20 
[    6.568680]  [<ffffffffa0321418>] ? __xfs_buf_delwri_submit+0x168/0x250 [xfs] 
[    6.576659]  [<ffffffffa0321154>] xfs_buf_submit+0x64/0x1c0 [xfs] 
[    6.583476]  [<ffffffffa0321418>] __xfs_buf_delwri_submit+0x168/0x250 [xfs] 
[    6.591260]  [<ffffffffa0322141>] ? xfs_buf_delwri_submit+0x41/0xf0 [xfs] 
[    6.598852]  [<ffffffffa0322141>] xfs_buf_delwri_submit+0x41/0xf0 [xfs] 
[    6.606253]  [<ffffffffa034d889>] xlog_recover_commit_trans+0x199/0x270 [xfs] 
[    6.614234]  [<ffffffffa034d9f3>] xlog_recovery_process_trans+0x93/0xd0 [xfs] 
[    6.622212]  [<ffffffffa034da97>] xlog_recover_process_ophdr+0x67/0xd0 [xfs] 
[    6.630093]  [<ffffffffa034dba2>] xlog_recover_process_data+0xa2/0xd0 [xfs] 
[    6.637875]  [<ffffffffa034dc6f>] xlog_recover_process+0x9f/0xf0 [xfs] 
[    6.645163]  [<ffffffffa034e16d>] xlog_do_recovery_pass+0x4ad/0x620 [xfs] 
[    6.652740]  [<ffffffffa034e369>] xlog_do_log_recovery+0x89/0xd0 [xfs] 
[    6.660037]  [<ffffffffa034e3c6>] xlog_do_recover+0x16/0xe0 [xfs] 
[    6.666840]  [<ffffffffa034f31f>] xlog_recover+0xbf/0x190 [xfs] 
[    6.673451]  [<ffffffffa03423aa>] xfs_log_mount+0xea/0x2e0 [xfs] 
[    6.680168]  [<ffffffffa0338c28>] xfs_mountfs+0x518/0x890 [xfs] 
[    6.686786]  [<ffffffffa0326fd0>] ? xfs_filestream_get_parent+0x80/0x80 [xfs] 
[    6.694763]  [<ffffffffa0339ad9>] ? xfs_mru_cache_create+0x129/0x190 [xfs] 
[    6.702448]  [<ffffffffa033c1a3>] xfs_fs_fill_super+0x3b3/0x4d0 [xfs] 
[    6.709629]  [<ffffffff811fb8d0>] mount_bdev+0x1b0/0x1f0 
[    6.715569]  [<ffffffffa033bdf0>] ? xfs_parseargs+0xbe0/0xbe0 [xfs] 
[    6.722575]  [<ffffffffa033a305>] xfs_fs_mount+0x15/0x20 [xfs] 
[    6.729088]  [<ffffffff811fc069>] mount_fs+0x39/0x1b0 
[    6.734719]  [<ffffffff811a2015>] ? __alloc_percpu+0x15/0x20 
[    6.741029]  [<ffffffff81217d1f>] vfs_kern_mount+0x5f/0xf0 
[    6.747154]  [<ffffffff8121a27e>] do_mount+0x24e/0xaa0 
[    6.752890]  [<ffffffff8119c55b>] ? strndup_user+0x4b/0xa0 
[    6.759011]  [<ffffffff8121ab66>] SyS_mount+0x96/0xf0 
[    6.764652]  [<ffffffff816954c9>] system_call_fastpath+0x16/0x1b 
[    6.771368] XFS (sda9): Metadata corruption detected at xfs_dir3_block_write_verify+0xfd/0x110 [xfs], xfs_dir3_block block 0xfc9f880 
[    6.784666] XFS (sda9): Unmount and run xfs_repair 
[    6.790011] XFS (sda9): First 64 bytes of corrupted metadata buffer: 
[    6.797104] ffff88086b520000: 58 44 42 33 00 00 00 00 00 00 00 00 0f c9 f8 80  XDB3............ 
[    6.806814] ffff88086b520010: 00 00 00 00 00 00 00 00 6e c2 7e d0 a3 ac 4e e7  ........n.~...N. 
[    6.816524] ffff88086b520020: 85 5d c8 4c 86 ce f3 a3 00 00 00 00 10 05 36 df  .].L..........6. 
[    6.826232] ffff88086b520030: 01 08 00 18 02 90 00 18 02 d8 00 18 00 00 00 00  ................ 
[    6.835943] XFS (sda9): xfs_do_force_shutdown(0x8) called from line 1288 of file fs/xfs/xfs_buf.c.  Return address = 0xffffffffa031f811 
[    6.849531] XFS (sda9): Corruption of in-memory data detected.  Shutting down filesystem 
[    6.858564] XFS (sda9): Please umount the filesystem and rectify the problem(s) 
[    6.920989] XFS (sda9): log mount/recovery failed: error -117 
[    6.927558] XFS (sda9): log mount failed 
[      
FAILED   
] Failed to mo[    
  OK     
] Started Emergency Shell.  
         Starting Emergency Shell...  
[    
  OK     
] Reached target Emergency Mode.  
 
Generating "/run/initramfs/rdsosreport.txt" 
 
 
Entering emergency mode. Exit the shell to continue. 
Type "journalctl" to view system logs. 
You might want to save "/run/initramfs/rdsosreport.txt" to a USB stick or /boot 
after mounting them and attach it to a bug report. 
 
 
:/# [-- MARK -- Fri Jul 15 19:35:00 2016] 
[-- MARK -- Fri Jul 15 19:40:00 2016] 


Other job:
https://beaker.engineering.redhat.com/recipes/2872992
https://beaker.engineering.redhat.com/recipes/2872993


I will try to reproduce this problem by manual. If I can find root reason.

Comment 14 Dave Young 2016-10-11 08:26:13 UTC

*** This bug has been marked as a duplicate of bug 1375457 ***


Note You need to log in before you can comment on or make changes to this bug.