Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 1510585 - vdo creation fails on multiple partitioned devices
Summary: vdo creation fails on multiple partitioned devices
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: vdo
Version: 7.5
Hardware: x86_64
OS: Linux
unspecified
medium
Target Milestone: rc
: ---
Assignee: corwin
QA Contact: Jakub Krysl
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-11-07 17:46 UTC by Corey Marthaler
Modified: 2019-03-06 00:46 UTC (History)
8 users (show)

Fixed In Version: 6.1.0.44
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-04-10 15:47:46 UTC


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHEA-2018:0871 None None None 2018-04-10 15:48:08 UTC

Description Corey Marthaler 2017-11-07 17:46:33 UTC
Description of problem:

[root@harding-02 ~]# vdo create --name vdo_a1 --device /dev/mapper/mpatha1
Creating VDO vdo_a1
vdo: ERROR - vdoformat: physical size must be a multiple of block size 4096


Nov  7 11:40:34 harding-02 kernel: kvdo: modprobe: loaded version 
Nov  7 11:40:34 harding-02 vdo: ERROR - vdoformat: physical size must be a multiple of block size 4096
Nov  7 11:40:34 harding-02 systemd: Started /usr/sbin/lvm pvscan --cache 253:20.


Version-Release number of selected component (if applicable):
3.10.0-772.el7.x86_64

lvm2-2.02.176-2.el7    BUILT: Fri Nov  3 07:46:53 CDT 2017
lvm2-libs-2.02.176-2.el7    BUILT: Fri Nov  3 07:46:53 CDT 2017
lvm2-cluster-2.02.176-2.el7    BUILT: Fri Nov  3 07:46:53 CDT 2017
lvm2-lockd-2.02.176-2.el7    BUILT: Fri Nov  3 07:46:53 CDT 2017
lvm2-python-boom-0.8-2.el7    BUILT: Fri Nov  3 07:48:54 CDT 2017
cmirror-2.02.176-2.el7    BUILT: Fri Nov  3 07:46:53 CDT 2017
device-mapper-1.02.145-2.el7    BUILT: Fri Nov  3 07:46:53 CDT 2017
device-mapper-libs-1.02.145-2.el7    BUILT: Fri Nov  3 07:46:53 CDT 2017
device-mapper-event-1.02.145-2.el7    BUILT: Fri Nov  3 07:46:53 CDT 2017
device-mapper-event-libs-1.02.145-2.el7    BUILT: Fri Nov  3 07:46:53 CDT 2017
device-mapper-persistent-data-0.7.3-2.el7    BUILT: Tue Oct 10 04:00:07 CDT 2017
sanlock-3.5.0-1.el7    BUILT: Wed Apr 26 09:37:30 CDT 2017
sanlock-lib-3.5.0-1.el7    BUILT: Wed Apr 26 09:37:30 CDT 2017
vdo-6.1.0.34-8    BUILT: Fri Nov  3 06:58:45 CDT 2017
kmod-kvdo-6.1.0.34-7.el7    BUILT: Fri Nov  3 06:44:06 CDT 2017

Comment 3 Corey Marthaler 2017-11-07 19:02:12 UTC
# Works on base mpath volume:

[root@harding-02 ~]# vdo create --name vdo_a1 --device /dev/mapper/mpatha
Creating VDO vdo_a1
Starting VDO vdo_a1
Starting compression on VDO vdo_a1
VDO instance 0 volume is ready at /dev/mapper/vdo_a1

Comment 4 Mike Snitzer 2017-11-07 19:13:20 UTC
(In reply to Corey Marthaler from comment #0)
> Description of problem:
> 
> [root@harding-02 ~]# vdo create --name vdo_a1 --device /dev/mapper/mpatha1
> Creating VDO vdo_a1
> vdo: ERROR - vdoformat: physical size must be a multiple of block size 4096
> 
> 
> Nov  7 11:40:34 harding-02 kernel: kvdo: modprobe: loaded version 
> Nov  7 11:40:34 harding-02 vdo: ERROR - vdoformat: physical size must be a
> multiple of block size 4096
> Nov  7 11:40:34 harding-02 systemd: Started /usr/sbin/lvm pvscan --cache
> 253:20.

What did you use to create the partition?  Is the partition's start aligned on a 4K boundary?  If not that could be why vdoformat is failing.

Please share a dmsetup table output for the multipath device and the partition's linear (kpartx created?) DM device ontop.

Comment 5 Corey Marthaler 2017-11-07 19:19:40 UTC
It appears to have a problem w/ any device with multiple partitions (and nothing to do w/ mpath).


# Mpath
[root@harding-02 ~]# dmsetup table | grep mpathb
mpathb: 0 524288000 multipath 2 queue_if_no_path retain_attached_hw_handler 1 alua 2 1 service-time 0 2 2 65:64 1 1 8:192 1 1 service-time 0 2 2 65:192 1 1 8:64 1 1 
mpathb2: 0 262132605 linear 253:2 262148670
mpathb1: 0 262148607 linear 253:2 63
[root@harding-02 ~]# kpartx /dev/mapper/mpathb
mpathb1 : 0 262148607 /dev/mapper/mpathb 63
mpathb2 : 0 262132605 /dev/mapper/mpathb 262148670
[root@harding-02 ~]# vdo create --name foo --device /dev/mapper/mpathb1
Creating VDO foo
vdo: ERROR - vdoformat: physical size must be a multiple of block size 4096
[root@harding-02 ~]# vdo create --name foo --device /dev/mapper/mpathb2
Creating VDO foo
vdo: ERROR - vdoformat: physical size must be a multiple of block size 4096



# Phys device
[root@host-026 ~]# cat /proc/partitions | grep sda
   8        0   26214400 sda
   8        1   13109008 sda1
   8        2   13101007 sda2

[root@host-026 ~]# vdo create --name foo --device /dev/sda1
Creating VDO foo
vdo: ERROR - vdoformat: physical size must be a multiple of block size 4096

vdo: ERROR - vdoformat: physical size must be a multiple of block size 4096

[root@host-026 ~]# kpartx /dev/sda
sda1 : 0 26218017 /dev/sda 63
sda2 : 0 26202015 /dev/sda 26218080

Comment 6 Mike Snitzer 2017-11-07 19:32:18 UTC
(In reply to Corey Marthaler from comment #5)
> It appears to have a problem w/ any device with multiple partitions (and
> nothing to do w/ mpath).
> 
> 
> # Mpath
> [root@harding-02 ~]# dmsetup table | grep mpathb
> mpathb: 0 524288000 multipath 2 queue_if_no_path retain_attached_hw_handler
> 1 alua 2 1 service-time 0 2 2 65:64 1 1 8:192 1 1 service-time 0 2 2 65:192
> 1 1 8:64 1 1 
> mpathb2: 0 262132605 linear 253:2 262148670
> mpathb1: 0 262148607 linear 253:2 63
> [root@harding-02 ~]# kpartx /dev/mapper/mpathb
> mpathb1 : 0 262148607 /dev/mapper/mpathb 63
> mpathb2 : 0 262132605 /dev/mapper/mpathb 262148670

...

> # Phys device
> [root@host-026 ~]# cat /proc/partitions | grep sda
>    8        0   26214400 sda
>    8        1   13109008 sda1
>    8        2   13101007 sda2
> 

> [root@host-026 ~]# kpartx /dev/sda
> sda1 : 0 26218017 /dev/sda 63
> sda2 : 0 26202015 /dev/sda 26218080

You've created partitions (likely with fdisk?) that are using the old MSDOS layout where the first partition starts at sector 63.  parted should default to aligning the partition to 1MB unless the optimal_io_size et al published for the device warrant the parted tool to start the partitions at a different alignment.

Honestly this is user error.  (Mixed with our willingness to have fdisk default to creating partitions that are misaligned!!! that is the most alarming aspect of this report.. cannot believe linux partitioning tools still suck this badly).  Cc'ing kzak to get his take (again).

Comment 7 Sweet Tea Dorminy 2017-11-10 02:06:35 UTC
One could make a device of size, say, 128G+1K, I think, if one uses LVM with extent size 1K. A VDO on such a device would fail in a similar fashion without the use of fdisk.

Comment 9 Sweet Tea Dorminy 2017-11-15 04:09:44 UTC
The VDO side is done and merged (see fix version); the question is whether the comments here indicate other tools/bugs/etc. need work.

Comment 12 Jakub Krysl 2017-11-23 16:00:06 UTC
I reproduced this using fdisk on 512K block size on our storageqe-73 server:
# fdisk /dev/mapper/mpathb                                                          
--snip--                                                                                                                                                                                                                                                                                                                                                                                                  
Command (m for help): n                                                                                  
Partition type:                                                                                                                                                                   
   p   primary (0 primary, 0 extended, 4 free)                                                                                         
   e   extended                                                                                                                                                                                                   
Select (default p):                                                                                                                                                                                               
Using default response p                                                                                                                                                                                           
Partition number (1-4, default 1):                                                                                                                                                                                
First sector (2048-20971519, default 2048): 63                                                                                                   
Value out of range.                                                                                      
First sector (2048-20971519, default 2048): 2049                                                                                       
Last sector, +sectors or +size{K,M,G} (2049-20971519, default 20971519): 20971518                        
Partition 1 of type Linux and of size 10 GiB is set     

vdo-6.1.0.34-8:
# vdo create --name vdo --device /dev/mapper/mpathb1 
Creating VDO vdo
vdo: ERROR - vdoformat: physical size must be a multiple of block size 4096

vdo-6.1.0.55-10:
# vdo create --name vdo --device /dev/mapper/mpathb1 
Creating VDO vdo
Starting VDO vdo
Starting compression on VDO vdo
VDO instance 0 volume is ready at /dev/mapper/vdo

[ 1938.982240] uds: modprobe: loaded version 6.1.0.31                                                    
[ 1939.008869] kvdo: modprobe: loaded version 6.1.0.55                                                   
[ 1939.191978] kvdo0:dmsetup: starting device 'vdo' device instantiation 0 (ti=ffffbfbac0f39040) write policy sync                                                                                                 
[ 1939.237289] kvdo0:dmsetup: zones: 1 logical, 1 physical, 1 hash; base threads: 5                      
[ 1939.378236] kvdo0:dmsetup: Setting UDS index target state to online                                   
[ 1939.406448] kvdo0:dmsetup: uds: kvdo0:dedupeQ: loading or rebuilding index: dev=/dev/mapper/mpathb1 offset=4096 size=2781704192                                                                                 
[ 1939.407207] uds: kvdo0:dedupeQ: Failed loading or rebuilding index: UDS Error: No index found (1061)  
uds: kvdo0:dedupeQ: creating index: dev=/dev/mapper/mpathb1 offset=4096 size=2781704192                  
uds: kvdo0:dedupeQ: Using 6 indexing zones for concurrency.                                              

[ 1939.562739] device 'vdo' started                 
[ 1939.577886] device-mapper: table: 253:8: adding target device (start sect 0 len 20969464) caused an alignment inconsistency                                                                                     
[ 1939.627898] kvdo0:dmsetup: resuming device 'vdo' 
[ 1939.648616] kvdo0:dmsetup: device 'vdo' resumed  
[ 1939.692555] kvdo0:packerQ: compression is enabled

Please note the device-mapper alignment inconsistency warning. I believe this is to be expected and the presence of this warning is a nice feature. But to be sure this is OK I created filesystem, mounted and wrote some data from /dev/urandom.
I started getting I/O Buffer Errors on the VDO device. First using XFS (created with discards):
# dd if=/dev/urandom of=vdo/random count=10000 bs=1M status=progress                
8049917952 bytes (8.0 GB) copied, 64.411219 s, 125 MB/s                                                  
dd: error writing ‘vdo/random’: Input/output error  
7727+0 records in                                   
7726+0 records out                                                                                       
8101298176 bytes (8.1 GB) copied, 64.8483 s, 125 MB/s  

[ 2329.776346] Buffer I/O error on dev dm-8, logical block 1682150, lost async page write
[ 2329.819027] Buffer I/O error on dev dm-8, logical block 1682151, lost async page write
[ 2329.856170] Buffer I/O error on dev dm-8, logical block 1682152, lost async page write
[ 2329.892801] Buffer I/O error on dev dm-8, logical block 1682153, lost async page write
[ 2329.910608] Buffer I/O error on dev dm-8, logical block 1708774, lost async page write
[ 2329.910613] Buffer I/O error on dev dm-8, logical block 1708775, lost async page write
[ 2329.910616] Buffer I/O error on dev dm-8, logical block 1708776, lost async page write
[ 2329.910618] Buffer I/O error on dev dm-8, logical block 1708777, lost async page write
[ 2329.910620] Buffer I/O error on dev dm-8, logical block 1708778, lost async page write
[ 2329.910621] Buffer I/O error on dev dm-8, logical block 1708779, lost async page write
[ 2336.110791] buffer_io_error: 207590 callbacks suppressed
[ 2336.137096] Buffer I/O error on dev dm-8, logical block 1876199, lost async page write
[ 2336.176722] Buffer I/O error on dev dm-8, logical block 1876200, lost async page write
[ 2336.213219] Buffer I/O error on dev dm-8, logical block 1876201, lost async page write
[ 2336.249112] Buffer I/O error on dev dm-8, logical block 1876202, lost async page write
[ 2336.259195] Buffer I/O error on dev dm-8, logical block 1904871, lost async page write
[ 2336.259201] Buffer I/O error on dev dm-8, logical block 1904872, lost async page write
[ 2336.259203] Buffer I/O error on dev dm-8, logical block 1904873, lost async page write
[ 2336.259205] Buffer I/O error on dev dm-8, logical block 1904874, lost async page write
[ 2336.259207] Buffer I/O error on dev dm-8, logical block 1904875, lost async page write
[ 2336.259209] Buffer I/O error on dev dm-8, logical block 1904876, lost async page write
[ 2338.202447] XFS (dm-8): metadata I/O error: block 0x9ffc58 ("xlog_iodone") error 28 numblks 64
[ 2338.209229] XFS (dm-8): metadata I/O error: block 0x8 ("xfs_buf_iodone_callback_error") error 28 numblks 8
[ 2338.252338] XFS (dm-8): metadata I/O error: block 0x9ffc60 ("xlog_iodone") error 28 numblks 64
[ 2338.252370] XFS (dm-8): Log I/O Error Detected.  Shutting down filesystem
[ 2338.252371] XFS (dm-8): Please umount the filesystem and rectify the problem(s)
[ 2338.252376] XFS (dm-8): Failing async write on buffer block 0x20. Retrying async write.
[ 2338.252378] XFS (dm-8): Failing async write on buffer block 0x28. Retrying async write.
[ 2338.252378] XFS (dm-8): Failing async write on buffer block 0x8. Retrying async write.


So to eliminate error on XFS side, I tested with EXT4 (again created with discards):
# dd if=/dev/urandom of=vdo/random count=10000 bs=1M status=progress                
7377780736 bytes (7.4 GB) copied, 59.257049 s, 125 MB/s                                                  
dd: error writing ‘vdo/random’: Read-only file system                                                    
7143+0 records in                                   
7142+0 records out                                  
7488929792 bytes (7.5 GB) copied, 60.1745 s, 124 MB/s  

[ 2572.304654] EXT4-fs warning (device dm-8): ext4_end_bio:316: I/O error -28 writing to inode 12 (offset 6425673728 size 4489216 starting block 1749861)
[ 2572.372411] Buffer I/O error on device dm-8, logical block 1749861
[ 2572.400162] EXT4-fs warning (device dm-8): ext4_end_bio:316: I/O error -28 writing to inode 12 (offset 6425673728 size 4763648 starting block 1749862)
[ 2572.459905] Buffer I/O error on device dm-8, logical block 1749862
[ 2572.487692] EXT4-fs warning (device dm-8): ext4_end_bio:316: I/O error -28 writing to inode 12 (offset 6425673728 size 4763648 starting block 1749863)
[ 2572.548701] Buffer I/O error on device dm-8, logical block 1749863
[ 2572.576416] EXT4-fs warning (device dm-8): ext4_end_bio:316: I/O error -28 writing to inode 12 (offset 6425673728 size 4763648 starting block 1749864)
[ 2572.636868] Buffer I/O error on device dm-8, logical block 1749864
[ 2572.664199] EXT4-fs warning (device dm-8): ext4_end_bio:316: I/O error -28 writing to inode 12 (offset 6425673728 size 4763648 starting block 1749865)
[ 2572.724801] Buffer I/O error on device dm-8, logical block 1749865
[ 2572.752564] EXT4-fs warning (device dm-8): ext4_end_bio:316: I/O error -28 writing to inode 12 (offset 6425673728 size 4763648 starting block 1749866)
[ 2572.814471] Buffer I/O error on device dm-8, logical block 1749866
[ 2572.844917] EXT4-fs warning (device dm-8): ext4_end_bio:316: I/O error -28 writing to inode 12 (offset 6425673728 size 4763648 starting block 1749867)
[ 2572.907179] Buffer I/O error on device dm-8, logical block 1749867
[ 2572.934908] EXT4-fs warning (device dm-8): ext4_end_bio:316: I/O error -28 writing to inode 12 (offset 6425673728 size 4763648 starting block 1749868)
[ 2572.995345] Buffer I/O error on device dm-8, logical block 1749868
[ 2573.023158] EXT4-fs warning (device dm-8): ext4_end_bio:316: I/O error -28 writing to inode 12 (offset 6425673728 size 4763648 starting block 1749869)
[ 2573.083747] Buffer I/O error on device dm-8, logical block 1749869
[ 2573.111821] EXT4-fs warning (device dm-8): ext4_end_bio:316: I/O error -28 writing to inode 12 (offset 6425673728 size 4763648 starting block 1749870)
[ 2573.172606] Buffer I/O error on device dm-8, logical block 1749870
[ 2575.794617] JBD2: Detected IO errors while flushing file data on dm-8-8
[ 2575.828750] Aborting journal on device dm-8-8.
[ 2575.851007] buffer_io_error: 194548 callbacks suppressed
[ 2575.875206] Buffer I/O error on dev dm-8, logical block 1081344, lost sync page write
[ 2575.910402] JBD2: Error -5 detected when updating journal superblock for dm-8-8.
[ 2578.976738] Buffer I/O error on dev dm-8, logical block 0, lost sync page write
[ 2579.014578] EXT4-fs error (device dm-8): ext4_journal_check_start:56: Detected aborted journal
[ 2579.056630] EXT4-fs (dm-8): Remounting filesystem read-only
[ 2579.081702] EXT4-fs (dm-8): previous I/O error to superblock detected
[ 2579.111212] Buffer I/O error on dev dm-8, logical block 0, lost sync page write
[ 2579.143909] EXT4-fs (dm-8): ext4_writepages: jbd2_start: 57344 pages, ino 12; err -30

I managed to get these errors on other configurations as well. I am investigating that now and will probably create a new BZ when I have enough info and I am sure this is not related. Till than I leave this open.

Comment 13 Jakub Krysl 2017-11-28 08:35:05 UTC
Created BZ 1517911. As it is not related to this one (reproduced it on other setups), setting as verified.

Comment 16 errata-xmlrpc 2018-04-10 15:47:46 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2018:0871


Note You need to log in before you can comment on or make changes to this bug.