Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 1358267 - Wrong size and utilization of pool
Summary: Wrong size and utilization of pool
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Storage Console
Classification: Red Hat
Component: core
Version: 2
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
: 2
Assignee: anmol babu
QA Contact: Filip Balák
URL:
Whiteboard:
Depends On: 1360230 1377269
Blocks: 1346350 Console-2-Async 1357845
TreeView+ depends on / blocked
 
Reported: 2016-07-20 11:57 UTC by Martin Kudlej
Modified: 2016-10-19 15:20 UTC (History)
9 users (show)

Fixed In Version: rhscon-ceph-0.0.41-1.el7scon.x86_64
Doc Type: Bug Fix
Doc Text:
Previously, pools list in Console displayed incorrect storage utilization and capacity data due to multiple CRUSH hierarchies. Due to this: * Pool utilization values on the dashboard, clusters view and pool listing page displayed incorrectly. * No alerts was sent if the actual pool utilization surpasses the configured thresholds * False alerts might be generated for pool utilization With this update, the pools list in Console displays the correct storage utilization and capacity data.
Clone Of:
Environment:
Last Closed: 2016-10-19 15:20:24 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Red Hat Bugzilla 1340452 None CLOSED Most Used Pools list is not correct 2019-04-04 10:18:31 UTC
Red Hat Bugzilla 1355723 None CLOSED Pools list dashboard page provides incorrect storage utilization/capacity data 2019-04-04 10:18:31 UTC
Red Hat Bugzilla 1357436 None CLOSED cluster dashboard->most used profile wrong colour 2019-04-04 10:18:30 UTC
Red Hat Bugzilla 1357447 None CLOSED main dashboard utilization 2019-04-04 10:18:31 UTC
Red Hat Bugzilla 1359840 None CLOSED pool utilization alerts 2019-04-04 10:18:30 UTC
Red Hat Product Errata RHSA-2016:2082 normal SHIPPED_LIVE Moderate: Red Hat Storage Console 2 security and bug fix update 2017-04-18 19:29:02 UTC


Description Martin Kudlej 2016-07-20 11:57:21 UTC
Description of problem:
I have one cluster and one pool in it. There are 2 OSDs with 100GB each. Pool is set to be double replication.
Output from Ceph command is:

$ ceph -c /etc/ceph/qwerty.conf df
GLOBAL:
    SIZE     AVAIL      RAW USED     %RAW USED 
    199G     20297M         180G         90.08 
POOLS:
    NAME     ID     USED       %USED     MAX AVAIL     OBJECTS 
    onee     1      92160M     90.04        15222M          90 

Output from Console API is:
$ ./clusters.sh | jq .
{
  "storageid": "f989eaab-2550-4ab1-ae47-7b7dcf78f2c3",
  "name": "onee",
  "type": "",
  "tags": [],
  "clusterid": "a989abe7-5e29-4e47-a572-ce853150d94e",
  "size": "51GB",
  "status": 0,
  "replicas": 2,
  "profile": "default",
  "snapshots_enabled": false,
  "snapshot_schedule_ids": [],
  "quota_enabled": false,
  "quota_params": {},
  "options": {
    "crash_replay_interval": "0",
    "crush_ruleset": "0",
    "full": "false",
    "hashpspool": "true",
    "id": "1",
    "min_size": "1",
    "pg_num": "128",
    "pgp_num": "128"
  },
  "usage": {
    "used": 96636764160,
    "total": 112599164452,  <-- I think this is also wrong because it should be 100GB => 104857600

    "percentused": 85.82369561116536,   <--- This is wrong because it is not same utilization as in ceph command
    "updatedat": "2016-07-20 13:34:05.353204155 +0200 CEST"
  },
  "state": "",
  "almstatus": 4,
  "almwarncount": 1,
  "almcritcount": 0,
  "slus": [
    "0eb1c4d5-85dd-439e-a4fc-68bdd437a33f",
    "9a5d24d5-d119-481e-96cd-4c5e5e89a8ca",
    "3e666f4c-af05-4c5a-b242-4c41c6b03052"
  ]
}



Version-Release number of selected component (if applicable):
ceph-ansible-1.0.5-27.el7scon.noarch
ceph-installer-1.0.14-1.el7scon.noarch
rhscon-ceph-0.0.33-1.el7scon.x86_64
rhscon-core-0.0.34-1.el7scon.x86_64
rhscon-core-selinux-0.0.34-1.el7scon.noarch
rhscon-ui-0.0.48-1.el7scon.noarch

How reproducible:
100%

Steps to Reproduce:
1. create one cluster and one pool
2. copy some data into pool
3. check utilization graph in list of pools

Actual results:
Utilization of pool is wrong.

Expected results:
Pool utilization will be displayed on every place correctly.

Comment 2 Martin Kudlej 2016-07-21 12:08:39 UTC
I still see this with
ceph-ansible-1.0.5-31.el7scon.noarch
ceph-installer-1.0.14-1.el7scon.noarch
rhscon-ceph-0.0.34-1.el7scon.x86_64
rhscon-core-0.0.35-1.el7scon.x86_64
rhscon-core-selinux-0.0.35-1.el7scon.noarch
rhscon-ui-0.0.49-1.el7scon.noarch

./clusters.sh | jq .
[
  {
    "storageid": "00875c43-6221-4922-bc5a-e5499a7cae97",
    "name": "klkl",
    "type": "",
    "tags": [],
    "clusterid": "af45a20f-b0a6-4c6a-952a-9145715e53f4",
    "size": "5GB",
    "status": 0,
    "replicas": 2,
    "profile": "default",
    "snapshots_enabled": false,
    "snapshot_schedule_ids": [],
    "quota_enabled": false,
    "quota_params": {},
    "options": {
      "crash_replay_interval": "0",
      "crush_ruleset": "0",
      "full": "false",
      "hashpspool": "true",
      "id": "1",
      "min_size": "1",
      "pg_num": "128",
      "pgp_num": "128"
    },
    "usage": {
      "used": 9810477056,
      "total": 876523520,
      "percentused": 1119.2485805743124,
      "updatedat": "2016-07-21 14:01:53.359976493 +0200 CEST"
    },
    "state": "",
    "almstatus": 1,
    "almwarncount": 0,
    "almcritcount": 1,
    "slus": [
      "9ad26031-c48c-44d5-94b7-4fea54f0a18a",
      "c2b85ce3-eaff-4b7c-8b3a-74da1216056f"
    ]
  }
]

Comment 3 Lubos Trilety 2016-07-25 13:48:35 UTC
Tested on:
rhscon-core-0.0.36-1.el7scon.x86_64
rhscon-ui-0.0.50-1.el7scon.noarch
rhscon-ceph-0.0.36-1.el7scon.x86_64
rhscon-core-selinux-0.0.36-1.el7scon.noarch

{
  "storageid": "f268a5d8-66be-42f2-af31-f16dd6f7c55f",
  "name": "default",
  "type": "",
  "tags": [],
  "clusterid": "712ba28f-9840-4b5d-b19f-2ef2631ee762",
  "size": "5GB",
  "status": 0,
  "replicas": 4,
  "profile": "default",
  "snapshots_enabled": false,
  "snapshot_schedule_ids": [],
  "quota_enabled": false,
  "quota_params": {},
  "options": {
    "crash_replay_interval": "0",
    "crush_ruleset": "0",
    "full": "false",
    "hashpspool": "true",
    "id": "2",
    "min_size": "2",
    "pg_num": "128",
    "pgp_num": "128"
  },
  "usage": {
    "used": 5242880000,
    "total": 12015363917,
    "percentused": 43.63479987969473,
    "updatedat": "2016-07-25 15:30:28.211639431 +0200 CEST"
  },
  "state": "",
  "almstatus": 4,
  "almwarncount": 1,
  "almcritcount": 0,
  "slus": [
    "99ff63c0-974c-4737-b016-c4d4ef8488f8",
    "cf944995-4ccd-4311-be8a-551519120e9c",
    "4c5f6d14-68be-430c-8e35-75907fef7bbe",
    "32b6f092-e5c7-4c9f-9f1a-053040371c4e",
    "e71cd79a-1f7b-4803-a442-4267322599ac"
  ]
}


GLOBAL:
    SIZE       AVAIL      RAW USED     %RAW USED 
    92060M     68733M       23327M         25.34 
POOLS:
    NAME        ID     USED      %USED     MAX AVAIL     OBJECTS 
    ectest      1      2000M      3.26        18385M           2 
    default     2      5000M     21.72         6458M           5

Comment 4 Darshan 2016-07-26 10:22:46 UTC
We depend on json format output of "ceph df" command and that does not have the information about % used. So we calculate it in skyring as follows:

Percent Used = (USED * 100) / (USED + MAX AVAIL) 

But this value doesn't match with the one provided by the ceph df output. There is a confusion around how to calculate the utilization percent of a pool.

Have raised a Bug: https://bugzilla.redhat.com/show_bug.cgi?id=1360230  in ceph for getting clarity about this.

Comment 6 Martin Bukatovic 2016-07-26 12:51:57 UTC
If you decide to reuse standard output of `ceph df`, you myn need to tweak the
template displayed in the pool list.

Right now, the template from the pools list states something like:

> 2.0 GB out of 10.0 GB used

While the `ceph df` would show:

~~~
# ceph --cluster beta df
GLOBAL:
    SIZE       AVAIL      RAW USED     %RAW USED 
    20457M     16292M        4165M         20.36 
POOLS:
    NAME             ID     USED      %USED     MAX AVAIL     OBJECTS 
    beta_01_pool     1      2048M     20.02         8146M           2
~~~

So if you decide to reuse `ceph df` completely, you need to rework the
message, because showing eg. `2048M of 8146M` would not make sense.

Comment 7 Lubos Trilety 2016-07-26 13:08:40 UTC
(In reply to Lubos Trilety from comment #5)
> BTW
> 'rados df' give better information about the pool then 'cepf df'
> # rados df -p default
> pool name                 KB      objects       clones     degraded     
> unfound           rd        rd KB           wr        wr KB
> default              5242880            5            0            0         
> 0            0            0         1280      5242880
>   total used        21258036            5
>   total avail       62537772
>   total space       83795808
> 
> I am pretty sure we could use used, avail and total space without any
> problem.

With that said the percentage of used make sense
In my case it was:
# ceph df
GLOBAL:
    SIZE       AVAIL      RAW USED     %RAW USED 
    81831M     61072M       20759M         25.37 
POOLS:
    NAME        ID     USED      %USED     MAX AVAIL     OBJECTS 
    default     1      5120M     25.03         5072M           5

However the total space is not correct though as it seems all disks are counted not just those which are part of the used hierarchy. In my case I have pool with replication set to 4 and default storage profile with 4 osds.

# ceph osd tree
ID  WEIGHT  TYPE NAME                     UP/DOWN REWEIGHT PRIMARY-AFFINITY 
-11 0.00999 root test                                                       
-10 0.00999     host dhcp-126-101-test                                      
  0 0.00999         osd.0                      up  1.00000          1.00000 
 -9 0.02998 root ec_test                                                    
 -6 0.00999     host dhcp-126-103-ec_test                                   
  4 0.00999         osd.4                      up  1.00000          1.00000 
 -7 0.00999     host dhcp-126-102-ec_test                                   
  3 0.00999         osd.3                      up  1.00000          1.00000 
 -8 0.00999     host dhcp-126-105-ec_test                                   
  6 0.00999         osd.6                      up  1.00000          1.00000 
 -1 0.03998 root default                                                    
 -2 0.00999     host dhcp-126-101                                           
  1 0.00999         osd.1                      up  1.00000          1.00000 
 -3 0.00999     host dhcp-126-102                                           
  2 0.00999         osd.2                      up  1.00000          1.00000 
 -4 0.00999     host dhcp-126-103                                           
  5 0.00999         osd.5                      up  1.00000          1.00000 
 -5 0.00999     host dhcp-126-105                                           
  7 0.00999         osd.7                      up  1.00000          1.00000

All my osds have 10GB of space. So I agree that what usm is telling is more correct then what the ceph says. From that point of view the issue is more on the ceph side than on usm side.

Comment 8 anmol babu 2016-07-29 12:02:51 UTC
We now pick the utilization from the normal cli command ceph df instead of its  
json form. So the percentage used and used size is now the same as what the cli returns
and the total calculation is as follows(as per sjust's suggestions):
POOL TOTAL SIZE = MAX AVAIL / (.01 * (100 - %USED))
where MAX AVAIL and %USED are directly from ceph df cli o/p.

Comment 10 Lubos Trilety 2016-08-02 10:01:10 UTC
%USED doesn't show how much of the space for the pool is used, but how much of the space of cluster is used in the pool. With that said if I have several hierarchies the number is not correct. (At least not correct for calculating total size of pool from it.)

Reopening BZ 1360230

Comment 11 Lubos Trilety 2016-08-02 10:24:15 UTC
(In reply to anmol babu from comment #8)
> We now pick the utilization from the normal cli command ceph df instead of
> its  
> json form. So the percentage used and used size is now the same as what the
> cli returns
> and the total calculation is as follows(as per sjust's suggestions):
> POOL TOTAL SIZE = MAX AVAIL / (.01 * (100 - %USED))
> where MAX AVAIL and %USED are directly from ceph df cli o/p.

BTW your calculation for POOL TOTAL SIZE is wrong as MAX AVAIL counts with other pools as well. E.g. I have two pools in the same hierarchy (the same storage profile) and I fill one pool with some data MAX AVAIL for the other pool will be changed as well, because free space in the hierarchy for both pools is the same.

My configuration:
10GB OSDs, two pools with the same replication 4, one without replication
One pool is filled with some data.

# ceph df
GLOBAL:
    SIZE       AVAIL      RAW USED     %RAW USED 
    81831M     52873M       28957M         35.39 
POOLS:
    NAME          ID     USED      %USED     MAX AVAIL     OBJECTS 
    testpool      1      7168M     35.04         3023M           7 
    testpool2     4          0         0         3023M           0
    norep         2          0         0        12093M           0

As you can see MAX AVAIL was changed for all of them, not just the one with some data.

Comment 12 Lubos Trilety 2016-08-02 10:27:15 UTC
I forgot to add that in my configuration pools are created on default hierarchy with 4 OSDs.

# ceph osd tree
ID  WEIGHT  TYPE NAME                    UP/DOWN REWEIGHT PRIMARY-AFFINITY 
-11 0.00999 root test                                                      
-10 0.00999     host dhcp-126-101-test                                     
  0 0.00999         osd.0                     up  1.00000          1.00000 
 -9 0.02998 root ectest                                                    
 -6 0.00999     host dhcp-126-105-ectest                                   
  6 0.00999         osd.6                     up  1.00000          1.00000 
 -7 0.00999     host dhcp-126-102-ectest                                   
  2 0.00999         osd.2                     up  1.00000          1.00000 
 -8 0.00999     host dhcp-126-103-ectest                                   
  4 0.00999         osd.4                     up  1.00000          1.00000 
 -1 0.03998 root default                                                   
 -2 0.00999     host dhcp-126-101                                          
  1 0.00999         osd.1                     up  1.00000          1.00000 
 -3 0.00999     host dhcp-126-102                                          
  3 0.00999         osd.3                     up  1.00000          1.00000 
 -4 0.00999     host dhcp-126-103                                          
  5 0.00999         osd.5                     up  1.00000          1.00000 
 -5 0.00999     host dhcp-126-105                                          
  7 0.00999         osd.7                     up  1.00000          1.00000

Comment 15 anmol babu 2016-08-10 06:10:38 UTC
Looks good to me.

Comment 16 Nishanth Thomas 2016-09-14 13:21:41 UTC
Tested with latest ceph builds(ceph-mon-10.2.2-39.el7cp.x86_64) and it works as expected. Dependant bug(https://bugzilla.redhat.com/show_bug.cgi?id=1360230) also marked as verified

Comment 17 Filip Balák 2016-09-26 12:27:03 UTC
Tested with
Server:
ceph-ansible-1.0.5-33.el7scon.noarch
ceph-installer-1.0.15-2.el7scon.noarch
graphite-web-0.9.12-8.1.el7.noarch
rhscon-ceph-0.0.42-1.el7scon.x86_64
rhscon-core-selinux-0.0.43-1.el7scon.noarch
rhscon-core-0.0.43-1.el7scon.x86_64
rhscon-ui-0.0.57-1.el7scon.noarch

Node:
calamari-server-1.4.8-1.el7cp.x86_64
ceph-base-10.2.2-41.el7cp.x86_64
ceph-common-10.2.2-41.el7cp.x86_64
ceph-mon-10.2.2-41.el7cp.x86_64
ceph-osd-10.2.2-41.el7cp.x86_64
ceph-selinux-10.2.2-41.el7cp.x86_64
libcephfs1-10.2.2-41.el7cp.x86_64
python-cephfs-10.2.2-41.el7cp.x86_64
rhscon-agent-0.0.19-1.el7scon.noarch
rhscon-core-selinux-0.0.43-1.el7scon.noarch

and it works as it is expected. --> Verified

Comment 18 Anjana Suparna Sriram 2016-10-14 12:42:56 UTC
Hi Anmol,

I have edited the doc text for this bug. Kindly review and approve the text to be included in the async errata.

Regards,
Anjana

Comment 20 anmol babu 2016-10-16 04:53:10 UTC
Looks good to me

Comment 21 errata-xmlrpc 2016-10-19 15:20:24 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2016:2082


Note You need to log in before you can comment on or make changes to this bug.