Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 1684823 - [GSS] OpenShift Registry out of space - Unable to prune
Summary: [GSS] OpenShift Registry out of space - Unable to prune
Keywords:
Status: ASSIGNED
Alias: None
Product: Red Hat Gluster Storage
Classification: Red Hat
Component: fuse
Version: rhgs-3.4
Hardware: x86_64
OS: Linux
urgent
urgent
Target Milestone: ---
: ---
Assignee: Amar Tumballi
QA Contact: Rahul Hinduja
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-03-02 19:17 UTC by Cal Calhoun
Modified: 2019-04-11 07:59 UTC (History)
16 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:
Target Upstream Version:
sankarshan: needinfo-
sankarshan: needinfo-


Attachments (Terms of Use)

Comment 16 amansan 2019-03-04 14:26:10 UTC
Hi

The customer is waiting for an update, do you need any additional information ? 

Regards

Alicia

Comment 18 amansan 2019-03-06 14:18:43 UTC
Heal just finished:

```
sh-4.2# gluster v status vol_7cedfda39e63fbb59c2998233091cb10
Status of volume: vol_7cedfda39e63fbb59c2998233091cb10
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Brick 10.28.139.73:/var/lib/heketi/mounts/v
g_e48e0dbb3dd65dd250ce3c997effe892/brick_34
934b39c2999e4f55a0c89aecab5718/brick        49161     0          Y       18510
Brick 10.28.139.13:/var/lib/heketi/mounts/v
g_329c82fd567da85c0b62c5dda1c342f3/brick_5d
1fcdf29e210fbf066c910126d43524/brick        49159     0          Y       28716
Brick 10.28.139.166:/var/lib/heketi/mounts/
vg_f2e34e863e3a99b21011b4d8344ec848/brick_4
cd6d96d7b40cd17b59c4950b28cf3bc/brick       49159     0          Y       50806
Brick 10.28.139.73:/var/lib/heketi/mounts/v
g_c7be81950c8c69dbb33fbbea58d5d3a5/brick_78
9c14aa897f8d1a75d0954ff2dd0227/brick        49161     0          Y       18510
Brick 10.28.139.166:/var/lib/heketi/mounts/
vg_042c5a884998313285e3d00c6c59470c/brick_4
65fda0ee5c13e5d78e3279461e1fe99/brick       49159     0          Y       50806
Brick 10.28.139.13:/var/lib/heketi/mounts/v
g_2b36d9ace572cb665ea1cb28ebe45081/brick_f5
7189b014ed23f279d80db687b89ad5/brick        49159     0          Y       28716
Brick 10.28.139.13:/var/lib/heketi/mounts/v
g_2b36d9ace572cb665ea1cb28ebe45081/brick_d8
a46b7c8bd752f2031577ea3be69112/brick        49159     0          Y       28716
Brick 10.28.139.166:/var/lib/heketi/mounts/
vg_b2b1c3052cd87d070a3ae7cd9188b653/brick_7
677d50724618caa89055fdd921bb501/brick       49159     0          Y       50806
Brick 10.28.139.73:/var/lib/heketi/mounts/v
g_d54f514570ec9f63a8d1e7b578b03059/brick_00
cd7128c612491996944fafb80bc8f9/brick        49161     0          Y       18510
Brick 10.28.139.73:/var/lib/heketi/mounts/v
g_c7be81950c8c69dbb33fbbea58d5d3a5/brick_5b
00db39119354483417a28b5d55aaf1/brick        49161     0          Y       18510
Brick 10.28.139.166:/var/lib/heketi/mounts/
vg_042c5a884998313285e3d00c6c59470c/brick_7
076b52b2f6219162e8973b338859c4e/brick       49159     0          Y       50806
Brick 10.28.139.13:/var/lib/heketi/mounts/v
g_2b36d9ace572cb665ea1cb28ebe45081/brick_99
7a09c1ac081ddc7e805ff759b7f29a/brick        49159     0          Y       28716
Brick 10.28.139.166:/var/lib/heketi/mounts/
vg_042c5a884998313285e3d00c6c59470c/brick_9
bd4899a61d0998ef986e5d714a0df4f/brick       49159     0          Y       50806
Brick 10.28.139.73:/var/lib/heketi/mounts/v
g_d54f514570ec9f63a8d1e7b578b03059/brick_b6
dd1cbab3c28cb84a56ce1d65093935/brick        49161     0          Y       18510
Brick 10.28.139.13:/var/lib/heketi/mounts/v
g_6eb35f532caca1f72b88a54475950a7d/brick_92
d7d2aca1a7d8b5c02b94f61c807120/brick        49159     0          Y       28716
Brick 10.28.139.166:/var/lib/heketi/mounts/
vg_b2b1c3052cd87d070a3ae7cd9188b653/brick_c
5a962f47423c1232d31e2c901d5de08/brick       49159     0          Y       50806
Brick 10.28.139.73:/var/lib/heketi/mounts/v
g_d54f514570ec9f63a8d1e7b578b03059/brick_2b
d672a66df2636fa91cc255cf4accf9/brick        49161     0          Y       18510
Brick 10.28.139.13:/var/lib/heketi/mounts/v
g_6eb35f532caca1f72b88a54475950a7d/brick_5f
2f7653e2732cdf542b8ac934e05352/brick        49159     0          Y       28716
Self-heal Daemon on localhost               N/A       N/A        Y       40089
Self-heal Daemon on storage02.appcanvas.net N/A       N/A        Y       97772
Self-heal Daemon on storage01.appcanvas.net N/A       N/A        Y       125437

Task Status of Volume vol_7cedfda39e63fbb59c2998233091cb10
------------------------------------------------------------------------------
There are no active volume tasks

sh-4.2# gluster v heal vol_7cedfda39e63fbb59c2998233091cb10 info
Brick 10.28.139.73:/var/lib/heketi/mounts/vg_e48e0dbb3dd65dd250ce3c997effe892/brick_34934b39c2999e4f55a0c89aecab5718/brick
Status: Connected
Number of entries: 0

Brick 10.28.139.13:/var/lib/heketi/mounts/vg_329c82fd567da85c0b62c5dda1c342f3/brick_5d1fcdf29e210fbf066c910126d43524/brick
Status: Connected
Number of entries: 0

Brick 10.28.139.166:/var/lib/heketi/mounts/vg_f2e34e863e3a99b21011b4d8344ec848/brick_4cd6d96d7b40cd17b59c4950b28cf3bc/brick
Status: Connected
Number of entries: 0

Brick 10.28.139.73:/var/lib/heketi/mounts/vg_c7be81950c8c69dbb33fbbea58d5d3a5/brick_789c14aa897f8d1a75d0954ff2dd0227/brick
Status: Connected
Number of entries: 0

Brick 10.28.139.166:/var/lib/heketi/mounts/vg_042c5a884998313285e3d00c6c59470c/brick_465fda0ee5c13e5d78e3279461e1fe99/brick
Status: Connected
Number of entries: 0

Brick 10.28.139.13:/var/lib/heketi/mounts/vg_2b36d9ace572cb665ea1cb28ebe45081/brick_f57189b014ed23f279d80db687b89ad5/brick
Status: Connected
Number of entries: 0

Brick 10.28.139.13:/var/lib/heketi/mounts/vg_2b36d9ace572cb665ea1cb28ebe45081/brick_d8a46b7c8bd752f2031577ea3be69112/brick
Status: Connected
Number of entries: 0

Brick 10.28.139.166:/var/lib/heketi/mounts/vg_b2b1c3052cd87d070a3ae7cd9188b653/brick_7677d50724618caa89055fdd921bb501/brick
Status: Connected
Number of entries: 0

Brick 10.28.139.73:/var/lib/heketi/mounts/vg_d54f514570ec9f63a8d1e7b578b03059/brick_00cd7128c612491996944fafb80bc8f9/brick
Status: Connected
Number of entries: 0

Brick 10.28.139.73:/var/lib/heketi/mounts/vg_c7be81950c8c69dbb33fbbea58d5d3a5/brick_5b00db39119354483417a28b5d55aaf1/brick
Status: Connected
Number of entries: 0

Brick 10.28.139.166:/var/lib/heketi/mounts/vg_042c5a884998313285e3d00c6c59470c/brick_7076b52b2f6219162e8973b338859c4e/brick
Status: Connected
Number of entries: 0

Brick 10.28.139.13:/var/lib/heketi/mounts/vg_2b36d9ace572cb665ea1cb28ebe45081/brick_997a09c1ac081ddc7e805ff759b7f29a/brick
Status: Connected
Number of entries: 0

Brick 10.28.139.166:/var/lib/heketi/mounts/vg_042c5a884998313285e3d00c6c59470c/brick_9bd4899a61d0998ef986e5d714a0df4f/brick
Status: Connected
Number of entries: 0

Brick 10.28.139.73:/var/lib/heketi/mounts/vg_d54f514570ec9f63a8d1e7b578b03059/brick_b6dd1cbab3c28cb84a56ce1d65093935/brick
Status: Connected
Number of entries: 0

Brick 10.28.139.13:/var/lib/heketi/mounts/vg_6eb35f532caca1f72b88a54475950a7d/brick_92d7d2aca1a7d8b5c02b94f61c807120/brick
Status: Connected
Number of entries: 0

Brick 10.28.139.166:/var/lib/heketi/mounts/vg_b2b1c3052cd87d070a3ae7cd9188b653/brick_c5a962f47423c1232d31e2c901d5de08/brick
Status: Connected
Number of entries: 0

Brick 10.28.139.73:/var/lib/heketi/mounts/vg_d54f514570ec9f63a8d1e7b578b03059/brick_2bd672a66df2636fa91cc255cf4accf9/brick
Status: Connected
Number of entries: 0

Brick 10.28.139.13:/var/lib/heketi/mounts/vg_6eb35f532caca1f72b88a54475950a7d/brick_5f2f7653e2732cdf542b8ac934e05352/brick
Status: Connected
Number of entries: 0

```

It shows correct here now, but only after killing 03 and forcing a heal:
```
[root@storage01 docker_registry]# ls -l docker/registry/v2/blobs/sha256/61/61cfb307586f943e554f5d9345c34f0865c0e2dac0f6e0d6f4b8549e67ee47a2
total 2
-rw-r--r--. 1 1000030000 2000 1992 Feb  7 12:02 data
[root@storage01 docker_registry]#
```

But not sure if there are any other files that could have been missed, curious if we can force a re-heal/rebalance or have a different way for OCS to double check everything within the volume.

Even though the split-brain for the data dir has gone, still issues as its now an empty file:

```
WMac5159685:~ c09010a$ docker pull docker-registry-default.apps.appcanvas.net/da-saas-images/software-app-saas@sha256:61cfb307586f943e554f5d9345c34f0865c0e2dac0f6e0d6f4b8549e67ee47a2
Error response from daemon: manifest for docker-registry-default.apps.appcanvas.net/da-saas-images/software-app-saas@sha256:61cfb307586f943e554f5d9345c34f0865c0e2dac0f6e0d6f4b8549e67ee47a2 not found
WMac5159685:~ c09010a$
```

File pretty much empty via FUSE:
```
[root@storage01 docker_registry]# cd docker/registry/v2/blobs/sha256/61/61cfb307586f943e554f5d9345c34f0865c0e2dac0f6e0d6f4b8549e67ee47a2
[root@storage01 61cfb307586f943e554f5d9345c34f0865c0e2dac0f6e0d6f4b8549e67ee47a2]# du -shc *
2.0K	data
2.0K	total
[root@storage01 61cfb307586f943e554f5d9345c34f0865c0e2dac0f6e0d6f4b8549e67ee47a2]#
```

storage01:
```
sh-4.2# for file in `gluster v info vol_7cedfda39e63fbb59c2998233091cb10 | grep 10.28.139.13 | awk -F ':' '{print $3}'`; do du -shc $file/docker/registry/v2/blobs/sha256/61/61cfb307586f943e554f5d9345c34f0865c0e2dac0f6e0d6f4b8549e67ee47a2; done
4.0K	/var/lib/heketi/mounts/vg_329c82fd567da85c0b62c5dda1c342f3/brick_5d1fcdf29e210fbf066c910126d43524/brick/docker/registry/v2/blobs/sha256/61/61cfb307586f943e554f5d9345c34f0865c0e2dac0f6e0d6f4b8549e67ee47a2
4.0K	total
0	/var/lib/heketi/mounts/vg_2b36d9ace572cb665ea1cb28ebe45081/brick_f57189b014ed23f279d80db687b89ad5/brick/docker/registry/v2/blobs/sha256/61/61cfb307586f943e554f5d9345c34f0865c0e2dac0f6e0d6f4b8549e67ee47a2
0	total
0	/var/lib/heketi/mounts/vg_2b36d9ace572cb665ea1cb28ebe45081/brick_d8a46b7c8bd752f2031577ea3be69112/brick/docker/registry/v2/blobs/sha256/61/61cfb307586f943e554f5d9345c34f0865c0e2dac0f6e0d6f4b8549e67ee47a2
0	total
0	/var/lib/heketi/mounts/vg_2b36d9ace572cb665ea1cb28ebe45081/brick_997a09c1ac081ddc7e805ff759b7f29a/brick/docker/registry/v2/blobs/sha256/61/61cfb307586f943e554f5d9345c34f0865c0e2dac0f6e0d6f4b8549e67ee47a2
0	total
0	/var/lib/heketi/mounts/vg_6eb35f532caca1f72b88a54475950a7d/brick_92d7d2aca1a7d8b5c02b94f61c807120/brick/docker/registry/v2/blobs/sha256/61/61cfb307586f943e554f5d9345c34f0865c0e2dac0f6e0d6f4b8549e67ee47a2
0	total
0	/var/lib/heketi/mounts/vg_6eb35f532caca1f72b88a54475950a7d/brick_5f2f7653e2732cdf542b8ac934e05352/brick/docker/registry/v2/blobs/sha256/61/61cfb307586f943e554f5d9345c34f0865c0e2dac0f6e0d6f4b8549e67ee47a2
0	total
sh-4.2#
```

storage02:
```
sh-4.2# for file in `gluster v info vol_7cedfda39e63fbb59c2998233091cb10 | grep 10.28.139.73 | awk -F ':' '{print $3}'`; do du -shc $file/docker/registry/v2/blobs/sha256/61/61cfb307586f943e554f5d9345c34f0865c0e2dac0f6e0d6f4b8549e67ee47a2; done
4.0K	/var/lib/heketi/mounts/vg_e48e0dbb3dd65dd250ce3c997effe892/brick_34934b39c2999e4f55a0c89aecab5718/brick/docker/registry/v2/blobs/sha256/61/61cfb307586f943e554f5d9345c34f0865c0e2dac0f6e0d6f4b8549e67ee47a2
4.0K	total
0	/var/lib/heketi/mounts/vg_c7be81950c8c69dbb33fbbea58d5d3a5/brick_789c14aa897f8d1a75d0954ff2dd0227/brick/docker/registry/v2/blobs/sha256/61/61cfb307586f943e554f5d9345c34f0865c0e2dac0f6e0d6f4b8549e67ee47a2
0	total
0	/var/lib/heketi/mounts/vg_d54f514570ec9f63a8d1e7b578b03059/brick_00cd7128c612491996944fafb80bc8f9/brick/docker/registry/v2/blobs/sha256/61/61cfb307586f943e554f5d9345c34f0865c0e2dac0f6e0d6f4b8549e67ee47a2
0	total
0	/var/lib/heketi/mounts/vg_c7be81950c8c69dbb33fbbea58d5d3a5/brick_5b00db39119354483417a28b5d55aaf1/brick/docker/registry/v2/blobs/sha256/61/61cfb307586f943e554f5d9345c34f0865c0e2dac0f6e0d6f4b8549e67ee47a2
0	total
0	/var/lib/heketi/mounts/vg_d54f514570ec9f63a8d1e7b578b03059/brick_b6dd1cbab3c28cb84a56ce1d65093935/brick/docker/registry/v2/blobs/sha256/61/61cfb307586f943e554f5d9345c34f0865c0e2dac0f6e0d6f4b8549e67ee47a2
0	total
0	/var/lib/heketi/mounts/vg_d54f514570ec9f63a8d1e7b578b03059/brick_2bd672a66df2636fa91cc255cf4accf9/brick/docker/registry/v2/blobs/sha256/61/61cfb307586f943e554f5d9345c34f0865c0e2dac0f6e0d6f4b8549e67ee47a2
0	total
sh-4.2#
```

storage03:
```
sh-4.2# for file in `gluster v info vol_7cedfda39e63fbb59c2998233091cb10 | grep 10.28.139.166 | awk -F ':' '{print $3}'`; do du -shc $file/docker/registry/v2/blobs/sha256/61/61cfb307586f943e554f5d9345c34f0865c0e2dac0f6e0d6f4b8549e67ee47a2; done
4.0K	/var/lib/heketi/mounts/vg_f2e34e863e3a99b21011b4d8344ec848/brick_4cd6d96d7b40cd17b59c4950b28cf3bc/brick/docker/registry/v2/blobs/sha256/61/61cfb307586f943e554f5d9345c34f0865c0e2dac0f6e0d6f4b8549e67ee47a2
4.0K	total
0	/var/lib/heketi/mounts/vg_042c5a884998313285e3d00c6c59470c/brick_465fda0ee5c13e5d78e3279461e1fe99/brick/docker/registry/v2/blobs/sha256/61/61cfb307586f943e554f5d9345c34f0865c0e2dac0f6e0d6f4b8549e67ee47a2
0	total
0	/var/lib/heketi/mounts/vg_b2b1c3052cd87d070a3ae7cd9188b653/brick_7677d50724618caa89055fdd921bb501/brick/docker/registry/v2/blobs/sha256/61/61cfb307586f943e554f5d9345c34f0865c0e2dac0f6e0d6f4b8549e67ee47a2
0	total
0	/var/lib/heketi/mounts/vg_042c5a884998313285e3d00c6c59470c/brick_7076b52b2f6219162e8973b338859c4e/brick/docker/registry/v2/blobs/sha256/61/61cfb307586f943e554f5d9345c34f0865c0e2dac0f6e0d6f4b8549e67ee47a2
0	total
0	/var/lib/heketi/mounts/vg_042c5a884998313285e3d00c6c59470c/brick_9bd4899a61d0998ef986e5d714a0df4f/brick/docker/registry/v2/blobs/sha256/61/61cfb307586f943e554f5d9345c34f0865c0e2dac0f6e0d6f4b8549e67ee47a2
0	total
0	/var/lib/heketi/mounts/vg_b2b1c3052cd87d070a3ae7cd9188b653/brick_c5a962f47423c1232d31e2c901d5de08/brick/docker/registry/v2/blobs/sha256/61/61cfb307586f943e554f5d9345c34f0865c0e2dac0f6e0d6f4b8549e67ee47a2
0	total
sh-4.2#
```

Yesterday however, I could pull the image:
```
$ time docker pull docker-registry-default.apps.appcanvas.net/da-saas-images/software-app-saas:saas-rc-190128-201902071645@sha256:61cfb307586f943e554f5d9345c34f0865c0e2dac0f6e0d6f4b8549e67ee47a2
sha256:61cfb307586f943e554f5d9345c34f0865c0e2dac0f6e0d6f4b8549e67ee47a2: Pulling from da-saas-images/software-app-saas
c325120ebc8d: Pull complete
c9d123037991: Pull complete
521a1d24b010: Pull complete
0e82bb9f4fba: Pull complete
aee3da753a34: Pull complete
f8dee6aae8b9: Pull complete
c147fa80b11a: Pull complete
a60ea87d6b32: Pull complete
Digest: sha256:61cfb307586f943e554f5d9345c34f0865c0e2dac0f6e0d6f4b8549e67ee47a2
Status: Downloaded newer image for docker-registry-default.apps.appcanvas.net/da-saas-images/software-app-saas@sha256:61cfb307586f943e554f5d9345c34f0865c0e2dac0f6e0d6f4b8549e67ee47a2

real	5m2.891s
user	0m0.136s
sys	0m0.095s
$

I suppose this happened again after adding storage03, what could we need to have a better idea ?

Comment 26 Dan Yocum 2019-03-07 16:52:32 UTC
Csaba,

What's the state of the patch to resolve the df issue discussed by Sonal in https://bugzilla.redhat.com/show_bug.cgi?id=1684823#c17 ?

I'm just the TAM, not an engineer maintaining the code.

Dan

Comment 27 Amar Tumballi 2019-03-12 11:50:33 UTC
> What's the state of the patch to resolve the df issue discussed by Sonal in https://bugzilla.redhat.com/show_bug.cgi?id=1684823#c17 ?


Dan, this issue is resolved in version RHGS 3.4.1 (Batch Update 1).

Is there anything more we should be looking at in this bug?


Note You need to log in before you can comment on or make changes to this bug.