Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.

Bug 1354616

Summary: [RH Ceph 2.0 / 10.2.2-15] TestLibRBD.SnapCreateViaLockOwner failed in writeback cache
Product: Red Hat Ceph Storage Reporter: Vasu Kulkarni <vakulkar>
Component: RBDAssignee: Jason Dillaman <jdillama>
Status: CLOSED ERRATA QA Contact: Vasu Kulkarni <vakulkar>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 2.0CC: ceph-eng-bugs, hnallurv, kdreyer, vakulkar
Target Milestone: rc   
Target Release: 2.0   
Hardware: Unspecified   
OS: Unspecified   
Fixed In Version: RHEL: ceph-10.2.2-19.el7cp Ubuntu: ceph_10.2.2-15redhat1xenial Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-08-23 19:44:00 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description Vasu Kulkarni 2016-07-11 17:52:13 UTC
Description of problem:
During full rbd regression run on 10.2.2-15, librbd test failed for following test(writeback cache)

2016-07-10T18:10:50.250 INFO:tasks.workunit.client.0.clara005.stdout:[  FAILED  ] 1 test, listed below:
2016-07-10T18:10:50.251 INFO:tasks.workunit.client.0.clara005.stdout:[  FAILED  ] TestLibRBD.SnapCreateViaLockOwner


2016-07-10T18:10:40.213 INFO:tasks.workunit.client.0.clara005.stdout:[       OK ] TestJournalReplay.SnapProtect (2033 ms)
2016-07-10T18:10:40.215 INFO:tasks.workunit.client.0.clara005.stdout:[ RUN      ] TestJournalReplay.SnapUnprotect
2016-07-10T18:10:42.243 INFO:tasks.workunit.client.0.clara005.stdout:[       OK ] TestJournalReplay.SnapUnprotect (2029 ms)
2016-07-10T18:10:42.244 INFO:tasks.workunit.client.0.clara005.stdout:[ RUN      ] TestJournalReplay.SnapRename
2016-07-10T18:10:43.360 INFO:tasks.workunit.client.0.clara005.stdout:[       OK ] TestJournalReplay.SnapRename (1117 ms)
2016-07-10T18:10:43.361 INFO:tasks.workunit.client.0.clara005.stdout:[ RUN      ] TestJournalReplay.SnapRollback
2016-07-10T18:10:44.366 INFO:tasks.workunit.client.0.clara005.stdout:[       OK ] TestJournalReplay.SnapRollback (1006 ms)
2016-07-10T18:10:44.367 INFO:tasks.workunit.client.0.clara005.stdout:[ RUN      ] TestJournalReplay.SnapRemove
2016-07-10T18:10:48.255 INFO:tasks.workunit.client.0.clara005.stdout:[       OK ] TestJournalReplay.SnapRemove (3889 ms)
2016-07-10T18:10:48.256 INFO:tasks.workunit.client.0.clara005.stdout:[ RUN      ] TestJournalReplay.Rename
2016-07-10T18:10:48.459 INFO:tasks.workunit.client.0.clara005.stdout:[       OK ] TestJournalReplay.Rename (205 ms)
2016-07-10T18:10:48.461 INFO:tasks.workunit.client.0.clara005.stdout:[ RUN      ] TestJournalReplay.Resize
2016-07-10T18:10:48.647 INFO:tasks.workunit.client.0.clara005.stdout:[       OK ] TestJournalReplay.Resize (187 ms)
2016-07-10T18:10:48.648 INFO:tasks.workunit.client.0.clara005.stdout:[ RUN      ] TestJournalReplay.Flatten
2016-07-10T18:10:49.556 INFO:tasks.workunit.client.0.clara005.stdout:[       OK ] TestJournalReplay.Flatten (909 ms)
2016-07-10T18:10:49.557 INFO:tasks.workunit.client.0.clara005.stdout:[ RUN      ] TestJournalReplay.ObjectPosition
2016-07-10T18:10:49.727 INFO:tasks.workunit.client.0.clara005.stdout:[       OK ] TestJournalReplay.ObjectPosition (171 ms)
2016-07-10T18:10:50.245 INFO:tasks.workunit.client.0.clara005.stdout:[----------] 13 tests from TestJournalReplay (17622 ms total)
2016-07-10T18:10:50.246 INFO:tasks.workunit.client.0.clara005.stdout:
2016-07-10T18:10:50.247 INFO:tasks.workunit.client.0.clara005.stdout:[----------] Global test environment tear-down
2016-07-10T18:10:50.248 INFO:tasks.workunit.client.0.clara005.stdout:[==========] 166 tests from 10 test cases ran. (326152 ms total)
2016-07-10T18:10:50.249 INFO:tasks.workunit.client.0.clara005.stdout:[  PASSED  ] 165 tests.
2016-07-10T18:10:50.250 INFO:tasks.workunit.client.0.clara005.stdout:[  FAILED  ] 1 test, listed below:
2016-07-10T18:10:50.251 INFO:tasks.workunit.client.0.clara005.stdout:[  FAILED  ] TestLibRBD.SnapCreateViaLockOwner

Version-Release number of selected component (if applicable):
10.2.2-15.el7cp (60cd52496ca02bdde9c2f4191e617f75166d87b6)

How reproducible:

Steps to Reproduce:
1. Run rbd/librbd/{cache/writeback.yaml cachepool/small.yaml clusters/{fixed-3.yaml openstack.yaml} copy-on-read/off.yaml fs/xfs.yaml msgr-failures/few.yaml workloads/c_api_tests_with_journaling.yaml

Additional info:

Full logs:

Comment 2 Jason Dillaman 2016-07-11 17:56:29 UTC
@Vasu: please include the full output from the fail test.

Comment 3 Vasu Kulkarni 2016-07-11 18:01:40 UTC

Do you want to use this full logs from here instead, its too many lines?

Full logs:

Comment 4 Jason Dillaman 2016-07-11 18:21:35 UTC
@Vasu: the test case will output what line had a failure -- you cut out that detail from the test's console output.

Comment 5 Vasu Kulkarni 2016-07-11 18:28:49 UTC

below is the line
2016-07-10T18:06:27.771 INFO:tasks.workunit.client.0.clara005.stdout:[ RUN      ] TestLibRBD.SnapCreateViaLockOwner
2016-07-10T18:06:27.772 INFO:tasks.workunit.client.0.clara005.stdout:using new format!
2016-07-10T18:06:28.250 INFO:tasks.workunit.client.0.clara005.stdout:test/librbd/ Failure

Comment 11 Vasu Kulkarni 2016-07-19 19:07:28 UTC
Verified in 10.2.2-22

Comment 13 errata-xmlrpc 2016-08-23 19:44:00 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.