Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.

Bug 1363949

Summary: [RFE][RADOS]:- When possible indicate which shard is inconsistent from the list-inconsistent-obj output
Product: Red Hat Ceph Storage Reporter: shylesh <shmohan>
Component: RADOSAssignee: David Zafman <dzafman>
Status: CLOSED ERRATA QA Contact: shylesh <shmohan>
Severity: medium Docs Contact: Erin Donnelly <edonnell>
Priority: unspecified    
Version: 2.0CC: ceph-eng-bugs, ceph-qe-bugs, dzafman, edonnell, hnallurv, icolle, jdurgin, kchai, kdreyer, pbyregow, sjust
Target Milestone: rcKeywords: FutureFeature
Target Release: 2.3   
Hardware: x86_64   
OS: Unspecified   
Whiteboard:
Fixed In Version: RHEL: ceph-10.2.7-2.el7cp Ubuntu: ceph_10.2.7-3redhat1xenial Doc Type: Known Issue
Doc Text:
.The `rados list-inconsistent-obj` command does not highlight inconsistent shards when it could have The output of the `rados list-inconsistent-obj` command does not explicitly show which shard is inconsistent when it could have.
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-06-19 13:26:16 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Bug Depends On:    
Bug Blocks: 1322504, 1383917, 1412948    

Description shylesh 2016-08-04 06:24:12 UTC
Description of problem:


Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1. I created an ecpool of count 3+2
2. created few objects and took 3 snapshots . After each snapshot i wrote some data
3. i picked one of the shard and corrupted user.ceph.snapset xattr
4. after running the scrub on primary , status is reporting an inconsistent object but list-inconsistent-obj doesn't tell which shard is actually having the problem.
 
 
 

Additional info:

    [root@magna105 ~]# rados list-inconsistent-obj 58.0
    [{"object":{"name":"obj988","nspace":"","locator":"","snap":"head"},"missing":false,"stat_err":false,"read_err":false,"data_digest_mismatch":false,
    "omap_digest_mismatch":false,"size_mismatch":false,"attr_mismatch":true,"shards":[{"osd":0,"missing":false,"read_error":false,"data_digest_mismatch":false,"omap_digest_mismatch":false,"size_mismatch":false,"data_digest_mismatch_oi":false,"omap_digest_mismatch_oi":false,"size_mismatch_oi":false,"size":1376,"attrs":{"attr":{"name":"_","value":"DwjrAAAABAMnAAAAAAAAAAYAAABvYmo5ODj+\/\/\/\/\/\/\/\/\/yLj0iEAAAAAADoAAAAAAAAABgMcAAAAOgAAAAAAAAD\/\/\/\/\/AAAAAAAAAAD\/\/\/\/\/\/\/\/\/\/wAAAABwEwAAAAAAAFVIAQCgCwAAAAAAAFRIAQACAhUAAAAI\/rUGAAAAAAABAAAAAAAAAAAAAACCAgAAAAAAAP7lnldiKDcpAgIVAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAcBMAAAAAAAAAAAAAAAAAAAA0AAAA\/uWeV3RxlikMgVqc\/\/\/\/\/w=="},"attr":{"name":"hinfo_key","value":"AQEgAAAAYAUAAAAAAAAFAAAAxBQDO+gZjinoGY4pxBQDO8QUAzs="},"attr":{"name":"snapset","value":"AgJxAAAAAgAAAAAAAAABAgAAAAIAAAAAAAAAAQAAAAAAAAACAAAAAQAAAAAAAAACAAAAAAAAAAIAAAABAAAAAAAAAAAAAAACAAAAAAAAAAAAAAACAAAAAQAAAAAAAACCAgAAAAAAAAIAAAAAAAAAggIAAAAAAAA="}}},{"osd":2,"missing":false,"read_error":false,"data_digest_mismatch":false,"omap_digest_mismatch":false,"size_mismatch":false,"data_digest_mismatch_oi":false,"omap_digest_mismatch_oi":false,"size_mismatch_oi":false,"size":1376,"attrs":{"attr":{"name":"_","value":"DwjrAAAABAMnAAAAAAAAAAYAAABvYmo5ODj+\/\/\/\/\/\/\/\/\/yLj0iEAAAAAADoAAAAAAAAABgMcAAAAOgAAAAAAAAD\/\/\/\/\/AAAAAAAAAAD\/\/\/\/\/\/\/\/\/\/wAAAABwEwAAAAAAAFVIAQCgCwAAAAAAAFRIAQACAhUAAAAI\/rUGAAAAAAABAAAAAAAAAAAAAACCAgAAAAAAAP7lnldiKDcpAgIVAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAcBMAAAAAAAAAAAAAAAAAAAA0AAAA\/uWeV3RxlikMgVqc\/\/\/\/\/w=="},"attr":{"name":"hinfo_key","value":"AQEgAAAAYAUAAAAAAAAFAAAAxBQDO+gZjinoGY4pxBQDO8QUAzs="},"attr":{"name":"snapset","value":"AgJxAAAAAgAAAAAAAAABAgAAAAIAAAAAAAAAAQAAAAAAAAACAAAAAQAAAAAAAAACAAAAAAAAAAIAAAABAAAAAAAAAAAAAAACAAAAAAAAAAAAAAACAAAAAQAAAAAAAACCAgAAAAAAAAIAAAAAAAAAggIAAAAAAAA="}}},{"osd":3,"missing":false,"read_error":false,"data_digest_mismatch":false,"omap_digest_mismatch":false,"size_mismatch":false,"data_digest_mismatch_oi":false,"omap_digest_mismatch_oi":false,"size_mismatch_oi":false,"size":1376,"attrs":{"attr":{"name":"_","value":"DwjrAAAABAMnAAAAAAAAAAYAAABvYmo5ODj+\/\/\/\/\/\/\/\/\/yLj0iEAAAAAADoAAAAAAAAABgMcAAAAOgAAAAAAAAD\/\/\/\/\/AAAAAAAAAAD\/\/\/\/\/\/\/\/\/\/wAAAABwEwAAAAAAAFVIAQCgCwAAAAAAAFRIAQACAhUAAAAI\/rUGAAAAAAABAAAAAAAAAAAAAACCAgAAAAAAAP7lnldiKDcpAgIVAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAcBMAAAAAAAAAAAAAAAAAAAA0AAAA\/uWeV3RxlikMgVqc\/\/\/\/\/w=="},"attr":{"name":"hinfo_key","value":"AQEgAAAAYAUAAAAAAAAFAAAAxBQDO+gZjinoGY4pxBQDO8QUAzs="},"attr":{"name":"snapset","value":"AgJxAAAAAgAAAAAAAAABAgAAAAIAAAAAAAAAAQAAAAAAAAACAAAAAQAAAAAAAAACAAAAAAAAAAIAAAABAAAAAAAAAAAAAAACAAAAAAAAAAAAAAACAAAAAQAAAAAAAACCAgAAAAAAAAIAAAAAAAAAggIAAAAAAAA="}}},{"osd":4,"missing":false,"read_error":false,"data_digest_mismatch":false,"omap_digest_mismatch":false,"size_mismatch":false,"data_digest_mismatch_oi":false,"omap_digest_mismatch_oi":false,"size_mismatch_oi":false,"size":1376,"attrs":{"attr":{"name":"_","value":"DwjrAAAABAMnAAAAAAAAAAYAAABvYmo5ODj+\/\/\/\/\/\/\/\/\/yLj0iEAAAAAADoAAAAAAAAABgMcAAAAOgAAAAAAAAD\/\/\/\/\/AAAAAAAAAAD\/\/\/\/\/\/\/\/\/\/wAAAABwEwAAAAAAAFVIAQCgCwAAAAAAAFRIAQACAhUAAAAI\/rUGAAAAAAABAAAAAAAAAAAAAACCAgAAAAAAAP7lnldiKDcpAgIVAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAcBMAAAAAAAAAAAAAAAAAAAA0AAAA\/uWeV3RxlikMgVqc\/\/\/\/\/w=="},"attr":{"name":"hinfo_key","value":"AQEgAAAAYAUAAAAAAAAFAAAAxBQDO+gZjinoGY4pxBQDO8QUAzs="},"attr":{"name":"snapset","value":"AgJxAAAAAgAAAAAAAAABAgAAAAIAAAAAAAAAAQAAAAAAAAACAAAAAQAAAAAAAAACAAAAAAAAAAIAAAABAAAAAAAAAAAAAAACAAAAAAAAAAAAAAACAAAAAQAAAAAAAACCAgAAAAAAAAIAAAAAAAAAggIAAAAAAAA="}}},{"osd":7,"missing":false,"read_error":false,"data_digest_mismatch":false,"omap_digest_mismatch":false,"size_mismatch":false,"data_digest_mismatch_oi":false,"omap_digest_mismatch_oi":false,"size_mismatch_oi":false,"size":1376,"attrs":{"attr":{"name":"_","value":"DwjrAAAABAMnAAAAAAAAAAYAAABvYmo5ODj+\/\/\/\/\/\/\/\/\/yLj0iEAAAAAADoAAAAAAAAABgMcAAAAOgAAAAAAAAD\/\/\/\/\/AAAAAAAAAAD\/\/\/\/\/\/\/\/\/\/wAAAABwEwAAAAAAAFVIAQCgCwAAAAAAAFRIAQACAhUAAAAI\/rUGAAAAAAABAAAAAAAAAAAAAACCAgAAAAAAAP7lnldiKDcpAgIVAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAcBMAAAAAAAAAAAAAAAAAAAA0AAAA\/uWeV3RxlikMgVqc\/\/\/\/\/w=="},"attr":{"name":"hinfo_key","value":"AQEgAAAAYAUAAAAAAAAFAAAAxBQDO+gZjinoGY4pxBQDO8QUAzs="},"attr":{"name":"snapset","value":"SEVMTE8="}}}]}]



From the above output its not possible to tell which shard is inconsistent. However user can go through value of each attr for each shard and can find it out which is cumbersome. It would be nice if list-inconsistent-obj itself can highlight the defective shard.

Comment 2 Samuel Just 2016-08-04 14:26:48 UTC
David's working on this as we speak for Kraken.

Comment 3 David Zafman 2016-08-07 00:51:08 UTC
The current in progress code for Kraken will give an "attr_value_mismatch" for this error and will indicate it on a particular shard.

I can see by a quick visual inspection that the snapset on osd 7 doesn't look like the others.

The scrub code picks an authoritative copy based on the integrity of certain fields.  In the reporter's example if osd 7 were to be selected as the authoritative copy the result will mark all the other OSDs with the "attr_value_mismatch" error.

Comment 12 David Zafman 2017-04-12 00:00:02 UTC
This was back ported after all (https://github.com/ceph/ceph/pull/13146).  I changed to 2.3 target release.

Comment 19 Parikshith 2017-05-24 10:39:41 UTC
[root@banshee 1.38s3_head]# rados  list-inconsistent-obj 1.38 --format=json-pretty
{
    "epoch": 163,
    "inconsistents": [
        {
            "object": {
                "name": "benchmark_data_aircobra.lab.eng.blr.redhat.c_79300_object79",
                "nspace": "",
                "locator": "",
                "snap": "head",
                "version": 2
            },
            "errors": [
                "attr_value_mismatch"
            ],
            "union_shard_errors": [],
            "selected_object_info": "1:1d3dd9ca:::benchmark_data_aircobra.lab.eng.blr.redhat.c_79300_object79:head(165'2 client.4724.0:80 dirty|data_digest|omap_digest s 4198176 uv 2 dd 52fd006a od ffffffff)",
            "shards": [
                {
                    "osd": 1,
                    "shard": 0,
                    "errors": [],
                    "size": 1399392,
                    "attrs": [
                        {
                            "name": "_",
                            "value": "DwggAQAABANcAAAAAAAAADsAAABiZW5jaG1hcmtfZGF0YV9haXJjb2JyYS5sYWIuZW5nLmJsci5yZWRoYXQuY183OTMwMF9vYmplY3Q3Of7\/\/\/\/\/\/\/\/\/uLybUwAAAAAAAQAAAAAAAAAGAxwAAAABAAAAAAAAAP\/\/\/\/8AAAAAAAAAAP\/\/\/\/\/\/\/\/\/\/AAAAAAIAAAAAAAAApQAAAAAAAAAAAAAAAAAAAAICFQAAAAh0EgAAAAAAAFAAAAAAAAAAAAAAACAPQAAAAAAAaB0lWbJfPAICAhUAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAACAAAAAAAAAAAAAAAAAAAAADQAAABoHSVZfJ\/TAmoA\/VL\/\/\/\/\/",
                            "Base64": true
                        },
                        {
                            "name": "hinfo_key",
                            "value": "AQEgAAAAYFoVAAAAAAAFAAAAr+GE2KhJYfGoSWHxr+GE2Kw7MEA=",
                            "Base64": true
                        },
                        {
                            "name": "snapset",
                            "value": "AgIZAAAAAAAAAAAAAAABAAAAAAAAAAAAAAAAAAAAAA==",
                            "Base64": true
                        }
                    ]
                },
                {
                    "osd": 2,
                    "shard": 2,
                    "errors": [],
                    "size": 1399392,
                    "attrs": [
                        {
                            "name": "_",
                            "value": "DwggAQAABANcAAAAAAAAADsAAABiZW5jaG1hcmtfZGF0YV9haXJjb2JyYS5sYWIuZW5nLmJsci5yZWRoYXQuY183OTMwMF9vYmplY3Q3Of7\/\/\/\/\/\/\/\/\/uLybUwAAAAAAAQAAAAAAAAAGAxwAAAABAAAAAAAAAP\/\/\/\/8AAAAAAAAAAP\/\/\/\/\/\/\/\/\/\/AAAAAAIAAAAAAAAApQAAAAAAAAAAAAAAAAAAAAICFQAAAAh0EgAAAAAAAFAAAAAAAAAAAAAAACAPQAAAAAAAaB0lWbJfPAICAhUAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAACAAAAAAAAAAAAAAAAAAAAADQAAABoHSVZfJ\/TAmoA\/VL\/\/\/\/\/",
                            "Base64": true
                        },
                        {
                            "name": "hinfo_key",
                            "value": "AQEgAAAAYFoVAAAAAAAFAAAAr+GE2KhJYfGoSWHxr+GE2Kw7MEA=",
                            "Base64": true
                        },
                        {
                            "name": "snapset",
                            "value": "AgIZAAAAAAAAAAAAAAABAAAAAAAAAAAAAAAAAAAAAA==",
                            "Base64": true
                        }
                    ]
                },
                {
                    "osd": 3,
                    "shard": 4,
                    "errors": [],
                    "size": 1399392,
                    "attrs": [
                        {
                            "name": "_",
                            "value": "DwggAQAABANcAAAAAAAAADsAAABiZW5jaG1hcmtfZGF0YV9haXJjb2JyYS5sYWIuZW5nLmJsci5yZWRoYXQuY183OTMwMF9vYmplY3Q3Of7\/\/\/\/\/\/\/\/\/uLybUwAAAAAAAQAAAAAAAAAGAxwAAAABAAAAAAAAAP\/\/\/\/8AAAAAAAAAAP\/\/\/\/\/\/\/\/\/\/AAAAAAIAAAAAAAAApQAAAAAAAAAAAAAAAAAAAAICFQAAAAh0EgAAAAAAAFAAAAAAAAAAAAAAACAPQAAAAAAAaB0lWbJfPAICAhUAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAACAAAAAAAAAAAAAAAAAAAAADQAAABoHSVZfJ\/TAmoA\/VL\/\/\/\/\/",
                            "Base64": true
                        },
                        {
                            "name": "hinfo_key",
                            "value": "AQEgAAAAYFoVAAAAAAAFAAAAr+GE2KhJYfGoSWHxr+GE2Kw7MEA=",
                            "Base64": true
                        },
                        {
                            "name": "snapset",
                            "value": "AgIZAAAAAAAAAAAAAAABAAAAAAAAAAAAAAAAAAAAAA==",
                            "Base64": true
                        }
                    ]
                },
                {
                    "osd": 4,
                    "shard": 3,
                    "errors": [],
                    "size": 1399392,
                    "attrs": [
                        {
                            "name": "_",
                            "value": "DwggAQAABANcAAAAAAAAADsAAABiZW5jaG1hcmtfZGF0YV9haXJjb2JyYS5sYWIuZW5nLmJsci5yZWRoYXQuY183OTMwMF9vYmplY3Q3Of7\/\/\/\/\/\/\/\/\/uLybUwAAAAAAAQAAAAAAAAAGAxwAAAABAAAAAAAAAP\/\/\/\/8AAAAAAAAAAP\/\/\/\/\/\/\/\/\/\/AAAAAAIAAAAAAAAApQAAAAAAAAAAAAAAAAAAAAICFQAAAAh0EgAAAAAAAFAAAAAAAAAAAAAAACAPQAAAAAAAaB0lWbJfPAICAhUAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAACAAAAAAAAAAAAAAAAAAAAADQAAABoHSVZfJ\/TAmoA\/VL\/\/\/\/\/",
                            "Base64": true
                        },
                        {
                            "name": "hinfo_key",
                            "value": "AQEgAAAAYFoVAAAAAAAFAAAAr+GE2KhJYfGoSWHxr+GE2Kw7MEA=",
                            "Base64": true
                        },
                        {
                            "name": "snapset",
                            "value": "0000000xxxx000000000",
                            "Base64": false
                        }
                    ]
                },
                {
                    "osd": 8,
                    "shard": 1,
                    "errors": [],
                    "size": 1399392,
                    "attrs": [
                        {
                            "name": "_",
                            "value": "DwggAQAABANcAAAAAAAAADsAAABiZW5jaG1hcmtfZGF0YV9haXJjb2JyYS5sYWIuZW5nLmJsci5yZWRoYXQuY183OTMwMF9vYmplY3Q3Of7\/\/\/\/\/\/\/\/\/uLybUwAAAAAAAQAAAAAAAAAGAxwAAAABAAAAAAAAAP\/\/\/\/8AAAAAAAAAAP\/\/\/\/\/\/\/\/\/\/AAAAAAIAAAAAAAAApQAAAAAAAAAAAAAAAAAAAAICFQAAAAh0EgAAAAAAAFAAAAAAAAAAAAAAACAPQAAAAAAAaB0lWbJfPAICAhUAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAACAAAAAAAAAAAAAAAAAAAAADQAAABoHSVZfJ\/TAmoA\/VL\/\/\/\/\/",
                            "Base64": true
                        },
                        {
                            "name": "hinfo_key",
                            "value": "AQEgAAAAYFoVAAAAAAAFAAAAr+GE2KhJYfGoSWHxr+GE2Kw7MEA=",
                            "Base64": true
                        },
                        {
                            "name": "snapset",
                            "value": "AgIZAAAAAAAAAAAAAAABAAAAAAAAAAAAAAAAAAAAAA==",
                            "Base64": true
                        }
                    ]
                }
            ]
        }
    ]
}

I am verifying this issue.
Errors section in individual shards/osds does't display any error. Am I missing something?

Comment 20 John Poelstra 2017-05-24 15:13:47 UTC
discussed at program meeting, need feedback from development today. gregory to followup

Comment 37 errata-xmlrpc 2017-06-19 13:26:16 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:1497