Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 1686827 - swift objects for glance consuming space on controllers
Summary: swift objects for glance consuming space on controllers
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-swift
Version: 10.0 (Newton)
Hardware: Unspecified
OS: Linux
Target Milestone: ---
: ---
Assignee: Pete Zaitcev
QA Contact: Mike Abrams
Depends On:
TreeView+ depends on / blocked
Reported: 2019-03-08 12:33 UTC by rohit londhe
Modified: 2019-04-09 05:26 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Last Closed:
Target Upstream Version:

Attachments (Terms of Use)
test.txt: List of Swift objects in one of controllers (deleted)
2019-03-12 23:52 UTC, Pete Zaitcev
no flags Details
image3.txt: List of Glance images (deleted)
2019-03-12 23:53 UTC, Pete Zaitcev
no flags Details
analysis script 1 (deleted)
2019-03-12 23:54 UTC, Pete Zaitcev
no flags Details

Description rohit londhe 2019-03-08 12:33:31 UTC
Description of problem:

swift objects for glance consuming space on controllers

1. There are 2 swift containers.

[root@controller-0 ~]# swift --os-auth-url $(hiera swift::proxy::authtoken::auth_uri) --os-username service:gnocchi --os-password $(hiera gnocchi::storage::swift::swift_key) list


[root@controller-0 ~]# swift --os-auth-url $(hiera swift::proxy::authtoken::auth_uri) --os-username service:glance --os-password $(hiera glance::backend::swift::swift_store_key) stat
                        Account: AUTH_d4d4bfeec4da4c40b37a951d27f0fce3
                     Containers: 2
                        Objects: 1034
                          Bytes: 204750736896
Containers in policy "policy-0": 2
   Objects in policy "policy-0": 1034
     Bytes in policy "policy-0": 204750736896
    X-Account-Project-Domain-Id: default
                    X-Timestamp: 1527452527.58882
                     X-Trans-Id: txa0d6b96d17df43b286502-005c809bf4
                   Content-Type: text/plain; charset=utf-8
                  Accept-Ranges: bytes
[root@controller-0 ~]#


[root@controller-0 ~]# swift --os-auth-url $(hiera swift::proxy::authtoken::auth_uri) --os-username service:gnocchi --os-password $(hiera gnocchi::storage::swift::swift_key) stat measure

         Account: AUTH_d4d4bfeec4da4c40b37a951d27f0fce3
       Container: measure
         Objects: 0
           Bytes: 0
        Read ACL:
       Write ACL:
         Sync To:
        Sync Key:
   Accept-Ranges: bytes
X-Storage-Policy: Policy-0
   Last-Modified: Mon, 04 Mar 2019 03:11:48 GMT
     X-Timestamp: 1527452527.61810
      X-Trans-Id: txbecf9d92a7aa460d85d1b-005c7fe180
    Content-Type: text/plain; charset=utf-8


Glance-images are 179.6G in size total and /srv/node/d1/objects are itself consuming 539G

[root@controller-2 d1]# du -sch *
2.1M    accounts
0       async_pending
1.1M    containers
539G    objects
16K     quarantined
0       tmp
539G    total

# for i in `find -L /srv/node/d1/objects/ -type f -name "*.data" `; do echo "$i"; swift-object-info -n $i|egrep "Path|Account|Container|Object"; done 

Path: /AUTH_d4d4bfeec4da4c40b37a951d27f0fce3/glance/b5c5b07d-0d25-4934-83d3-56a80a56bc4c-00017
  Account: AUTH_d4d4bfeec4da4c40b37a951d27f0fce3
  Container: glance
  Object: b5c5b07d-0d25-4934-83d3-56a80a56bc4c-00017
  Object hash: 9eb5fdd473f8bef817641e80e6068713

The entire output of all the objects has been attached.

If we reduce the image size from 539 - 179.6 = 359.4 

Then what is exactly utilizing 359.4 exactly, Need help in finding out.
Or what could be the size of glance objects if the images are 179.6 G

There are only 2 containers with Glance and Measure respectively.
Version-Release number of selected component (if applicable):

How reproducible:

Steps to Reproduce:

Actual results:
glance objects are consuming more space

Expected results:
It is expected that the size of glance objects should be somewhere near to the actual image size.

Additional info:

Background information: Initially, it was observed that-

1. Controller-1 was replaced with controller-3 but swift logs still show sync with old controller IP ''.
2. This issue is similar to what discussed in BZ[1] and the second workaround has been applied after which the issue seems to be resolved.
3. Sometime later we again observed this issue where old controller IP is noticed []

[root@controller-0 swift]# swift-ring-builder object.builder
object.builder, build version 6
1024 partitions, 3.000000 replicas, 1 regions, 1 zones, 4 devices, 0.00 balance, 0.00 dispersion
The minimum number of hours before a partition can be reassigned is 1 (0:00:00 remaining)
The overload factor is 0.00% (0.000000)
Ring file object.ring.gz is up-to-date
Devices:   id region zone     ip address:port replication ip:port  name weight partitions balance flags meta
            0      1    1    d1 100.00        768    0.00
            1      1    1    d1 100.00        768    0.00
            2      1    1    d1 100.00        768    0.00
            3      1    1    d1 100.00        768    0.00

AS per BZ [1 ]Fix has been provided in openstack-tripleo-common-5.4.7-1.el7ost

The Customer already seems to be with the latest version.

$ cat sos_commands/rpm/sh_-c_rpm_--nodigest_-qa_--qf_NAME_-_VERSION_-_RELEASE_._ARCH_INSTALLTIME_date_awk_-F_printf_-59s_s_n_1_2_sort_-V  | grep -i openstack-tripleo-common-

openstack-tripleo-common-5.4.7-6.el7ost.noarch              Tue Sep 25 08:07:05 2018


Comment 5 Pete Zaitcev 2019-03-12 23:52:18 UTC
Created attachment 1543441 [details]
test.txt: List of Swift objects in one of controllers

Comment 6 Pete Zaitcev 2019-03-12 23:53:49 UTC
Created attachment 1543442 [details]
image3.txt: List of Glance images

Comment 7 Pete Zaitcev 2019-03-12 23:54:30 UTC
Created attachment 1543443 [details]
analysis script 1

Note You need to log in before you can comment on or make changes to this bug.