Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 1353860 - lvconvert --splitmirrors and --splitcache fails
Summary: lvconvert --splitmirrors and --splitcache fails
Keywords:
Status: NEW
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: lvm2
Version: 7.3
Hardware: x86_64
OS: Linux
unspecified
unspecified
Target Milestone: rc
: ---
Assignee: David Teigland
QA Contact: cluster-qe@redhat.com
URL:
Whiteboard:
Depends On: 1354549
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-07-08 09:00 UTC by Roman Bednář
Modified: 2018-10-26 21:45 UTC (History)
7 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:
Target Upstream Version:


Attachments (Terms of Use)

Description Roman Bednář 2016-07-08 09:00:12 UTC
Description of problem:
lvconvert split functions do not work in the latest build. Regression, problem does not occur in 2.02.156.

How reproducible:
always

Steps to Reproduce:

 SCENARIO - [stacked_cache_origin_split_w_tracking_io_merge]
 Create a 3-way raid1, convert to cache origin with fs date, verify date, split image with tracking, change data on raid vol, merge split image data back, verify origin data
 virt-247: lvcreate  --type raid1 -m 2 -n split_origin -L 500M split_image /dev/sdf1 /dev/sdc1 /dev/sdd1

 Waiting until all mirror|raid volumes become fully syncd...
    0/1 mirror(s) are fully synced: ( 70.81% )
    1/1 mirror(s) are fully synced: ( 100.00% )
 Sleeping 15 sec

 virt-247: lvcreate  -n cache -L 500M split_image /dev/sdb1
 virt-247: lvcreate  -n cache_meta -L 8M split_image /dev/sdb1
 Create cache pool volume by combining the cache data and cache metadata (fast) volumes
 lvconvert --yes --type cache-pool --poolmetadata split_image/cache_meta split_image/cache
   WARNING: Converting logical volume split_image/cache and split_image/cache_meta to pool's data and metadata volumes.
   THIS WILL DESTROY CONTENT OF LOGICAL VOLUME (filesystem etc.)
 Create cached volume by combining the cache pool (fast) and origin (slow) volumes
 lvconvert --yes --type cache --cachepool split_image/cache split_image/split_origin

 Placing an ext filesystem on raid1 volume
 mke2fs 1.42.9 (28-Dec-2013)
 Mounting cache volume

 Writing files to /mnt/split_origin
 /usr/tests/sts-rhel7.3/bin/checkit -w /mnt/split_origin -f /tmp/split_originA.23923 -n 500

 Checking files on /mnt/split_origin
 /usr/tests/sts-rhel7.3/bin/checkit -w /mnt/split_origin -f /tmp/split_originA.23923 -v


 Issuing a sync to force data to disk
 splitting off leg from raid with tracking...
 virt-247: lvconvert --yes --splitmirrors 1 --trackchanges split_image/split_origin_corig /dev/sdf1
 couldn't split off raid
   Cannot convert internal LV split_image/split_origin_corig.


Tested with:
3.10.0-460.el7.x86_64

lvm2-2.02.160-1.el7    BUILT: Wed Jul  6 18:16:47 CEST 2016
lvm2-libs-2.02.160-1.el7    BUILT: Wed Jul  6 18:16:47 CEST 2016
lvm2-cluster-2.02.160-1.el7    BUILT: Wed Jul  6 18:16:47 CEST 2016
device-mapper-1.02.130-1.el7    BUILT: Wed Jul  6 18:16:47 CEST 2016
device-mapper-libs-1.02.130-1.el7    BUILT: Wed Jul  6 18:16:47 CEST 2016
device-mapper-event-1.02.130-1.el7    BUILT: Wed Jul  6 18:16:47 CEST 2016
device-mapper-event-libs-1.02.130-1.el7    BUILT: Wed Jul  6 18:16:47 CEST 2016
device-mapper-persistent-data-0.6.2-0.1.rc8.el7    BUILT: Wed May  4 09:56:34 CEST 2016
cmirror-2.02.160-1.el7    BUILT: Wed Jul  6 18:16:47 CEST 2016

Comment 1 Roman Bednář 2016-07-08 09:04:02 UTC
Adding reproducer for --splitcache


SCENARIO - [test_cache_create]
Test cache pool volume creation and combining cache data and cache metadata volumes
 
*** Cache info for this scenario ***
*  origin (slow):  /dev/sdf1
*  pool (fast):    /dev/sdg1
************************************
 
Adding "slow" and "fast" tags to corresponding pvs
Create origin (slow) volume
lvcreate  -L 4G -n corigin cache_sanity @slow
 
lvcreate  -L 2G -n test_cache cache_sanity /dev/sdg1
lvcreate  -L 8M -n test_cache_meta cache_sanity /dev/sdg1
 
1A. Test that cache pool volume creation works by combining the cache data and cache metadata (fast) volumes
lvconvert --yes --test --type cache-pool --poolmetadata cache_sanity/test_cache_meta cache_sanity/test_cache
  TEST MODE: Metadata will NOT be updated and volumes will not be (de)activated.
  WARNING: Converting logical volume cache_sanity/test_cache and cache_sanity/test_cache_meta to pool's data and metadata volumes.
  THIS WILL DESTROY CONTENT OF LOGICAL VOLUME (filesystem etc.)
 
1B. Test that cache pool volume creation doesn't work by combining a nonexistent cache data and cache metadata (fast) volumes
lvconvert --yes --test --type cache-pool --poolmetadata cache_sanity/test_cache_meta cache_sanity/FAKE
  TEST MODE: Metadata will NOT be updated and volumes will not be (de)activated.
  Failed to find logical volume "cache_sanity/FAKE"
 
Check that no cache pool volume was actually created when attempted with '--test'
 
2A. Test the origin volume can be cached with lvm auto creating a cash pool out of cache_sanity/test_cache
lvconvert --yes --test --type cache --cachepool cache_sanity/test_cache cache_sanity/corigin
  TEST MODE: Metadata will NOT be updated and volumes will not be (de)activated.
  WARNING: Converting logical volume cache_sanity/test_cache to pool's data volume.
  THIS WILL DESTROY CONTENT OF LOGICAL VOLUME (filesystem etc.)
 
2B. Test the origin volume can't be cached since we are attempting with a non existing cache pool
  TEST MODE: Metadata will NOT be updated and volumes will not be (de)activated.
  Unknown pool data LV FAKE_pool.
  Implicit conversion of --cachepool arg to type cache-pool failed.
 
Now actually create the cache pool by combining the cache data and cache metadata (fast) volumes
lvconvert --yes --type cache-pool --poolmetadata cache_sanity/test_cache_meta cache_sanity/test_cache
  WARNING: Converting logical volume cache_sanity/test_cache and cache_sanity/test_cache_meta to pool's data and metadata volumes.
  THIS WILL DESTROY CONTENT OF LOGICAL VOLUME (filesystem etc.)
 
3. Test the origin volume can be cached now since we have a proper cache pool (fast) volumes now
lvconvert --yes --test --type cache --cachepool cache_sanity/test_cache cache_sanity/corigin
  TEST MODE: Metadata will NOT be updated and volumes will not be (de)activated.
 
Test that the origin volume was not actually cached when attempted with '--test'
Now actually create the cached origin volume
 
Separating cache pool (lvconvert --splitcache) cache_sanity/test_cache from cache origin
  Cannot convert internal LV cache_sanity/test_cache.
couldn't split cache pool volume

===========================================
[root@host-082 ~]# lvs -a -o +devices
  LV                 VG            Attr       LSize   Pool         Origin Data%  Meta%  Move Log Cpy%Sync Convert Devices            
  corigin            cache_sanity  Cwi-a-C---   4.00g [test_cache]        0.00                   100.00           corigin_corig(0)  
  [corigin_corig]    cache_sanity  owi-aoC---   4.00g                                                             /dev/sdf1(0)      
  [lvol0_pmspare]    cache_sanity  ewi-------   8.00m                                                             /dev/sdg1(514)    
  [test_cache]       cache_sanity  Cwi---C---   2.00g                                                             test_cache_cdata(0)
  [test_cache_cdata] cache_sanity  Cwi-ao----   2.00g                                                             /dev/sdg1(0)      
  [test_cache_cmeta] cache_sanity  ewi-ao----   8.00m                                                             /dev/sdg1(512)    
 
[root@host-082 ~]# lvconvert --splitcache /dev/cache_sanity/test_cache
  Cannot convert internal LV cache_sanity/test_cache.

Comment 3 David Teigland 2016-07-08 18:50:20 UTC
Here's the "right" way to split a cache pool from a cache LV without having to use hidden LVs:

[root@null-01 lvm.git]# lvs -a test | grep cpool1
  [cpool1]           test Cwi---C---   1.00g                            
  [cpool1_cdata]     test Cwi-------   1.00g                            
  [cpool1_cmeta]     test ewi-------   1.00g                            
  lvol1              test Cri---C--- 100.00m [cpool1]                   

[root@null-01 lvm.git]# lvconvert --splitcache test/cpool1
  Cannot convert internal LV test/cpool1.

[root@null-01 lvm.git]# lvconvert --splitcache test/lvol1
  Logical volume test/lvol1 is not cached and cache pool test/cpool1 is unused.

[root@null-01 lvm.git]# lvs -a test | grep cpool1
  cpool1             test Cwi---C---   1.00g                           
  [cpool1_cdata]     test Cwi-------   1.00g                           
  [cpool1_cmeta]     test ewi-------   1.00g                           

I'd prefer to keep it this way, but if specifying the hidden cache pool rather than the cache LV was once an advertised method, then we can add an exception to allow it.  If we do allow it, can we exclude it from the list of advertised methods?

Comment 4 David Teigland 2016-07-08 18:57:10 UTC
> lvconvert --yes --splitmirrors 1 --trackchanges split_image/split_origin_corig /dev/sdf1
>   Cannot convert internal LV split_image/split_origin_corig.

The "right" way to do this is to run the command on the visible cache LV, not on the hidden _corig LV, i.e.

lvconvert --splitmirrors 1 VG/CacheLV

In this instance I'm even more skeptical that we advertised running the command on the hidden origin, but if we did then we can again add another exception to allow it.

Comment 5 David Teigland 2016-07-08 19:42:47 UTC
At this moment, the lvconvert code in git

allows --splitcache on a hidden/used cache pool
disallows --splitmirrors on a hidden/used cache pool

My suggestion is that we disallow both, and require these operations to be run on the visible cache LV (both of which currently work.)

Comment 6 Roman Bednář 2016-07-11 11:39:33 UTC
I'm not entirely sure that removing functionality that has been present until now is safe. But I agree with your approach to disable both for consistency, how about allowing both? I can't think of any good use case right now, but it would give users more control.

We will change the scenarios based on the outcome of this bug if needed.

Comment 7 Roman Bednář 2016-07-11 11:44:58 UTC
Additionally I found another issue with lvconvert. Segfaults when target PV is specified:
# lvs -a
  LV                   VG            Attr       LSize   Pool           Origin Data%  Meta%  Move Log Cpy%Sync Convert
  ...                                                          
  [lvol0_pmspare]      vg            ewi-------  12.00m                                                              
  mirror_lv            vg            rwi-a-r---   1.00g                                              100.00          
  [mirror_lv_rimage_0] vg            iwi-aor---   1.00g                                                              
  [mirror_lv_rimage_1] vg            iwi-aor---   1.00g                                                              
  [mirror_lv_rimage_2] vg            iwi-aor---   1.00g                                                              
  [mirror_lv_rmeta_0]  vg            ewi-aor---   4.00m                                                              
  [mirror_lv_rmeta_1]  vg            ewi-aor---   4.00m                                                              
  [mirror_lv_rmeta_2]  vg            ewi-aor---   4.00m                                                              


# lvconvert --yes --splitmirrors 1 --trackchanges vg/mirror_lv /dev/sdf 
Segmentation fault (core dumped)

# lvconvert --splitmirrors 1 vg/mirror_lv --name split_test --trackchanges
  mirror_lv_rimage_3 split from mirror_lv for read-only purposes.
  Use 'lvconvert --merge vg/mirror_lv_rimage_3' to merge back into mirror_lv


Valgrind

==13635== LEAK SUMMARY:
==13635==    definitely lost: 0 bytes in 0 blocks
==13635==    indirectly lost: 0 bytes in 0 blocks
==13635==      possibly lost: 0 bytes in 0 blocks
==13635==    still reachable: 1,592,035 bytes in 6,933 blocks
==13635==         suppressed: 0 bytes in 0 blocks
==13635== 
==13635== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 0 from 0)
==13635== 
==13635== 1 errors in context 1 of 1:
==13635== Invalid read of size 8
==13635==    at 0x1E4C89: lv_raid_split_and_track (in /usr/sbin/lvm)
==13635==    by 0x13AB3E: ??? (in /usr/sbin/lvm)
==13635==    by 0x13D6EE: ??? (in /usr/sbin/lvm)
==13635==    by 0x163335: process_each_lv_in_vg (in /usr/sbin/lvm)
==13635==    by 0x164638: process_each_lv (in /usr/sbin/lvm)
==13635==    by 0x140593: lvconvert (in /usr/sbin/lvm)
==13635==    by 0x14D8E8: lvm_run_command (in /usr/sbin/lvm)
==13635==    by 0x14E48F: lvm2_main (in /usr/sbin/lvm)
==13635==    by 0x5B4EB34: (below main) (in /usr/lib64/libc-2.17.so)
==13635==  Address 0x48 is not stack'd, malloc'd or (recently) free'd
==13635== 
==13635== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 0 from 0)
Segmentation fault (core dumped)

Comment 8 David Teigland 2016-07-11 14:08:29 UTC
(In reply to Roman Bednář from comment #7)
> Additionally I found another issue with lvconvert. Segfaults when target PV
> is specified:

This is a different issue and should be a separate bz.

Comment 9 Roman Bednář 2016-07-11 14:19:34 UTC
Ok, bug 1354549 has been created to track this issue separately.

Comment 10 David Teigland 2016-07-25 16:20:39 UTC
I'm going to let this sit for the time being, which means that --splitcache is the only operation you can be used on an attached/hidden cache pool.  Other commands like --splitmirrors need to be run on the cache volume.  We're going to be taking a much longer look at the lvconvert interface because there are considerable problems and inconsistencies with it, including the issues above.


Note You need to log in before you can comment on or make changes to this bug.