Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 1055944 - Online thin pool metadata resize corrupts metadata
Summary: Online thin pool metadata resize corrupts metadata
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: kernel
Version: 7.0
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: rc
: ---
Assignee: Mike Snitzer
QA Contact: yanfu,wang
URL:
Whiteboard:
Depends On:
Blocks: 986084
TreeView+ depends on / blocked
 
Reported: 2014-01-21 09:50 UTC by Zdenek Kabelac
Modified: 2014-06-18 06:49 UTC (History)
9 users (show)

Fixed In Version: 3.10.0-85.el7
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2014-06-13 10:21:27 UTC


Attachments (Terms of Use)

Description Zdenek Kabelac 2014-01-21 09:50:09 UTC
Description of problem:

The latest version of thin pool kernel target (1.10) is still incorrectly resizing metadata online with this result after the resize, when removal of thin volume from resized metadata is requested:

device-mapper: thin: 253:5: growing the metadata device from 512 to 852992 blocks
device-mapper: space map common: unable to decrement a reference count below 0
device-mapper: thin: 253:5: metadata operation 'dm_pool_commit_metadata' failed: error = -22
device-mapper: thin: 253:5: switching pool to read-only mode
device-mapper: space map common: dm_tm_shadow_block() failed
device-mapper: space map common: dm_tm_shadow_block() failed
device-mapper: space map metadata: unable to allocate new metadata block
device-mapper: thin: Deletion of thin device 2 failed.



Version-Release number of selected component (if applicable):
kernel 3.10.0-75.el7.x86_64

How reproducible:
For easier testing local build of lvm2 is needed with this small patch:

diff --git a/lib/thin/thin.c b/lib/thin/thin.c
index ff263d9..6011d77 100644
--- a/lib/thin/thin.c
+++ b/lib/thin/thin.c
@@ -618,7 +618,7 @@ static int _thin_target_present(struct cmd_context *cmd,
 		{ 1, 1, THIN_FEATURE_EXTERNAL_ORIGIN, "external_origin" },
 		{ 1, 4, THIN_FEATURE_BLOCK_SIZE, "block_size" },
 		{ 1, 5, THIN_FEATURE_DISCARDS_NON_POWER_2, "discards_non_power_2" },
-		{ 9, 9, THIN_FEATURE_METADATA_RESIZE, "metadata_resize" },
+		{ 1, 10, THIN_FEATURE_METADATA_RESIZE, "metadata_resize" },
 	};
 
 	static const char _lvmconf[] = "global/thin_disabled_features";


Steps to Reproduce:
1. Create pool with 2MB metadata and thin volumes
2. online resize metadata from 2MB to 256MB
3. Try to remove thin volumes

Actual results:


Expected results:


Additional info:

Comment 1 Joe Thornber 2014-01-21 09:55:43 UTC
I've just added a scenario to dmtest that reproduces this issue.

https://github.com/jthornber/device-mapper-test-suite/commit/1c2d7cd787c18753b6c2d7f4f486298f314a3c14


Run it using:

    dmtest run --suite thin-provisioning -n thin_remove_works_after_resize

Comment 3 Joe Thornber 2014-01-21 11:18:40 UTC
Fix here.

https://github.com/jthornber/linux-2.6/commit/36cde327b91e14df17ea374383e7735710d69ba6

We'll get a new brew build to you to test.

Comment 4 Mike Snitzer 2014-01-22 01:15:29 UTC
I've pushed the fix to the linux-dm.git 'for-next' branch for near-term inclusion in 3.14.

Comment 5 yanfu,wang 2014-02-18 10:00:57 UTC
(In reply to Joe Thornber from comment #1)
> I've just added a scenario to dmtest that reproduces this issue.
> 
> https://github.com/jthornber/device-mapper-test-suite/commit/
> 1c2d7cd787c18753b6c2d7f4f486298f314a3c14
> 
> 
> Run it using:
> 
>     dmtest run --suite thin-provisioning -n thin_remove_works_after_resize

Reproduced on kernel 3.10.0-75.el7.x86_64:
# dmtest run --suite thin-provisioning -n thin_remove_works_after_resize --profile spindle
Loaded suite thin-provisioning
Started
test_thin_remove_works_after_resize(MetadataResizeTests): E

Finished in 0.019723698 seconds.

  1) Error:
test_thin_remove_works_after_resize(MetadataResizeTests):
ProcessControl::ExitError: command failed: blockdev --getsz /dev/tsvg/lv1
    /root/device-mapper-test-suite/lib/dmtest/process.rb:96:in `wait'
    /root/device-mapper-test-suite/lib/dmtest/process.rb:111:in `really_run'
    /root/device-mapper-test-suite/lib/dmtest/process.rb:116:in `run_'
    /root/device-mapper-test-suite/lib/dmtest/process.rb:134:in `system'
    /root/device-mapper-test-suite/lib/dmtest/utils.rb:11:in `dev_size'
    /root/device-mapper-test-suite/lib/dmtest/utils.rb:23:in `_dd_device'
    /root/device-mapper-test-suite/lib/dmtest/utils.rb:45:in `wipe_device'
    /root/device-mapper-test-suite/lib/dmtest/thinp-mixin.rb:70:in `setup'
    /root/device-mapper-test-suite/lib/dmtest/tests/thin-provisioning/metadata_resize_tests.rb:22:in `setup'

1 tests, 0 assertions, 0 failures, 1 errors

Comment 6 yanfu,wang 2014-02-19 02:32:35 UTC
updated the reproduced test result:
# dmtest run --suite thin-provisioning -n thin_remove_works_after_resize --profile spindle
Loaded suite thin-provisioning
Started
test_thin_remove_works_after_resize(MetadataResizeTests): F

Finished in 0.898766328 seconds.

  1) Failure:
test_thin_remove_works_after_resize(MetadataResizeTests)
    [/usr/local/rvm/gems/ruby-1.9.3-p484/gems/rspec-expectations-2.14.5/lib/rspec/expectations/fail_with.rb:32:in `fail_with'
     /usr/local/rvm/gems/ruby-1.9.3-p484/gems/rspec-expectations-2.14.5/lib/rspec/expectations/handler.rb:36:in `handle_matcher'
     /usr/local/rvm/gems/ruby-1.9.3-p484/gems/rspec-expectations-2.14.5/lib/rspec/expectations/syntax.rb:53:in `should'
     /root/device-mapper-test-suite/lib/dmtest/tests/thin-provisioning/metadata_resize_tests.rb:257:in `block (2 levels) in test_thin_remove_works_after_resize'
     /root/device-mapper-test-suite/lib/dmtest/prelude.rb:6:in `bracket'
     /root/device-mapper-test-suite/lib/dmtest/device-mapper/lexical_operators.rb:12:in `with_dev'
     /root/device-mapper-test-suite/lib/dmtest/tests/thin-provisioning/metadata_resize_tests.rb:231:in `block in test_thin_remove_works_after_resize'
     /root/device-mapper-test-suite/lib/dmtest/device-mapper/lexical_operators.rb:37:in `call'
     /root/device-mapper-test-suite/lib/dmtest/device-mapper/lexical_operators.rb:37:in `block in with_devs'
     /root/device-mapper-test-suite/lib/dmtest/prelude.rb:6:in `bracket'
     /root/device-mapper-test-suite/lib/dmtest/device-mapper/lexical_operators.rb:32:in `with_devs'
     /root/device-mapper-test-suite/lib/dmtest/tests/thin-provisioning/metadata_resize_tests.rb:226:in `test_thin_remove_works_after_resize']:
expected: false value
     got: true

1 tests, 0 assertions, 1 failures, 0 errors

@Joe, is the same failure as your reproducer?

Comment 7 Linda Wang 2014-03-09 04:18:27 UTC
The patchset that reference in comment#3 is posted here:

http://post-office.corp.redhat.com/archives/rhkernel-list/2014-January/msg02798.html

and checked in under Bug 1057197:

* Mon Feb 03 2014 Jarod Wilson <jarod@redhat.com> [3.10.0-83.el7]
...
- [md] dm-log-userspace: allow mark requests to piggyback on flush requests (Mike Snitzer) [1057197]
- [md] dm-space-map-metadata: fix bug in resizing of thin metadata (Mike Snitzer) [1057197]
- [md] dm-cache: add policy name to status output (Mike Snitzer) [1057197]
- [md] dm-thin: fix pool feature parsing (Mike Snitzer) [1057197]
- [md] dm-sysfs: fix a module unload race (Mike Snitzer) [1057197]
- [md] dm-snapshot: use dm-bufio prefetch (Mike Snitzer) [1057197]
- [md] dm-snapshot: use dm-bufio (Mike Snitzer) [1057197]
- [md] dm-snapshot: prepare for switch to using dm-bufio (Mike Snitzer) [1057197]
- [md] dm-snapshot: use GFP_KERNEL when initializing exceptions (Mike Snitzer) [1057197]

and two follow on patches here:

* Fri Feb 07 2014 Jarod Wilson <jarod@redhat.com> [3.10.0-85.el7]
- [md] dm-cache: do not add migration to completed list before unhooking bio (Mike Snitzer) [1057197]
- [md] dm-cache: move hook_info into common portion of per_bio_data structure (Mike Snitzer) [1057197]

-----

therefore move this bug to MODIFIED and add to kernel erratum for
QE to verify.

Comment 10 Ludek Smid 2014-06-13 10:21:27 UTC
This request was resolved in Red Hat Enterprise Linux 7.0.

Contact your manager or support representative in case you have further questions about the request.


Note You need to log in before you can comment on or make changes to this bug.