Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 1367219 - docker-storage-setup causes docker to fail when auto-extend is set. [NEEDINFO]
Summary: docker-storage-setup causes docker to fail when auto-extend is set.
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: docker
Version: 7.2
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: rc
: ---
Assignee: Vivek Goyal
QA Contact: atomic-bugs@redhat.com
URL:
Whiteboard:
Depends On:
Blocks: 1420851
TreeView+ depends on / blocked
 
Reported: 2016-08-15 22:43 UTC by Ryan Howe
Modified: 2019-03-06 00:56 UTC (History)
8 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2017-08-15 18:47:59 UTC
vgoyal: needinfo? (rhowe)


Attachments (Terms of Use)

Description Ryan Howe 2016-08-15 22:43:27 UTC
Description of problem:

 docker-storage-setup causes docker to fail when auto-extend is set. docker-storage-setup tries to extend lv with no extents and fails causing docker to fail. 

Version-Release number of selected component (if applicable):

RHEL 7.2 

How reproducible:
50% 

Steps to Reproduce:
1. Set up docker storage 

VG=docker-vg

2. Fill up docker-storage where the lv tries to extend beyond max. 
3. Reboot node 
4. Docker fails to start and requires storage to be reconfigured. 

Actual results:

 - Docker fails to start after reboot because docker-storage-setup fails to extend due to 0 extents available. 


Expected results:

  - For the lv to stop extending. 


Additional info:

- errors
 docker-storage-setup[2519]: Volume group "vda1" not found
 docker-storage-setup[2519]: Cannot process volume group vda1
 docker-storage-setup[2519]: /dev/vdb has partitions: vdb1
 systemd[1]: docker-storage-setup.service: main process exited, code=exited, status=1/FAILURE
 systemd[1]: Failed to start Docker Storage Setup.
 systemd[1]: Unit docker-storage-setup.service entered failed state.
 systemd[1]: docker-storage-setup.service failed.
 systemd[1]: Starting Docker Application Container Engine...
 docker[2534]: time="2016-07-18T15:44:24.832001710-05:00" level=fatal msg="Error starting daemon: error initializing graphdriver: Unable to take ownership of thin-pool (docker--vg-docker--pool) that already has used data blocks"
 systemd[1]: docker.service: main process exited, code=exited, status=1/FAILURE
 systemd[1]: Failed to start Docker Application Container Engine.
 systemd[1]: Unit docker.service entered failed state.
 systemd[1]: docker.service failed.




 - workaround, disable auto_extend_pool and set data size to 99% because 100% will fail due to number of extents. 

# systemctl stop docker
# lvremove /dev/docker-vg/docker-pool
# lvremove /dev/docker-vg/docker-poolmeta # rm -rf /var/lib/docker/*

# vi /etc/sysconfig/docker-storage-setup
VG=docker-vg
AUTO_EXTEND_POOL=no
DATA_SIZE=99%VG

# docker-storage-setup
# systemctl start docker

Comment 6 Qian Cai 2017-01-27 17:10:32 UTC
A few observations:

Your d-s-s config "VG=docker-vg" which is not obviously to me if your docker thinp is backed by a separate block device or just rootfs. If it is rootfs, once it is full, everything is in a mess so docker daemon might will fail as expected. If it is possible, before you fill the docker thinp, show us the output of,

- docker info
- lvs
- pvdisplay
- lvdisplay 

Just to make sure the thinp is created properly and used by docker with a separate block device.

I won't be able to reproduce on the latest 7.3.2 yet.
docker-1.12.5-14.el7.x86_64

After fill the docker thinp backed by a separate block device, I did a reboot (there is an xfs hang task due to thinp full that prevents the shutdown but a different issue).

After reboot, the docker starts successfully and still use the thinp.
# docker info
Storage Driver: devicemapper
 Pool Name: docker--vg-docker--pool

# lvs
  LV          VG        Attr       LSize  Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  docker-pool docker-vg twi-a-t--- 58.39g             100.00 6.02 

# docker run rhel7 dd if=/dev/zero of=/dssfile bs=1M
/usr/bin/docker-current: Error response from daemon: devmapper: Thin Pool has 0 free data blocks which is less than minimum required 11958 free data blocks. Create more free space in thin pool or use dm.min_free_space option to change behavior.
See '/usr/bin/docker-current run --help'.

Comment 7 Qian Cai 2017-01-27 17:20:19 UTC
FYI, bz 1417257 is created to track the shutdown dockerd hang task issue.

Comment 8 Daniel Walsh 2017-06-30 15:57:12 UTC
Vivek is this fixed in container-storage-setup?

Comment 10 Vivek Goyal 2017-08-15 18:40:43 UTC
Is this problem still reproducible?

I don't understand the original problem to begin with. Based on original problem, following is the error message.

docker[2534]: time="2016-07-18T15:44:24.832001710-05:00" level=fatal msg="Error starting daemon: error initializing graphdriver: Unable to take ownership of thin-pool (docker--vg-docker--pool) that already has used data blocks"

So docker is complaining that it has been asked to setup things on an lvm thin pool which is not empty to begin with. This is saftey check in docker to make sure we don't take ownership of a thin pool which is being used for something else already.

Hence this sounds more like to me a configuration issue and either existing docker meatadata got deleted from /var/lib/docker/ dir or something else happened.

I can't see how it is related to pool being full. If there is no space in volume group, pool extension will fail, but docker should still start and continue to use same thin pool.

Comment 11 Vivek Goyal 2017-08-15 18:47:59 UTC
Ryan, I am closing this as NOTABUG. Because I think this is not related to extension of thin pool failing. Based on the error message in the comment 0, this is related to some misconfiguration which happened along the line.

If you can reproduce the issue, please re-open this bug.


Note You need to log in before you can comment on or make changes to this bug.