Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 1597797 - containers are not gracefully stopped on shutdown/reboot
Summary: containers are not gracefully stopped on shutdown/reboot
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: python-paunch
Version: 13.0 (Queens)
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: z2
: 13.0 (Queens)
Assignee: Alex Schultz
QA Contact: Artem Hrechanychenko
URL:
Whiteboard:
Depends On: 1595733
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-07-03 15:52 UTC by Alex Schultz
Modified: 2018-08-29 16:33 UTC (History)
11 users (show)

Fixed In Version: python-paunch-2.5.0-2.el7ost
Doc Type: If docs needed, set a value
Doc Text:
Clone Of: 1595733
Environment:
Last Closed: 2018-08-29 16:32:35 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Launchpad 1778913 None None None 2018-07-03 15:52:35 UTC
RDO 14585 None None None 2018-07-03 15:52:35 UTC
Red Hat Product Errata RHBA-2018:2573 None None None 2018-08-29 16:33:24 UTC

Description Alex Schultz 2018-07-03 15:52:36 UTC
+++ This bug was initially created as a clone of Bug #1595733 +++

There is no mechanism to gracefully shutdown containers when shutting down/rebooting a node. They keep running until systemd performs a process cleanup and sends SIGTERM to all running process. If they are still running for some interval after the SIGTERM, systemd sends SIGKILL. It is unknown what impact that this might have.

Example from console logs:

[ 1556.163360] type=1700 audit(1529955121.410:884): dev=qr-90058585-3f prom=0 old_prom=256 auid=4294967295 uid=0 gid=0 ses=4294967295
[ 1558.874838] systemd-shutdown[1]: Sending SIGKILL to remaining processes...
[ 1558.886653] systemd-shutdown[1]: Sending SIGKILL to PID 2044 (docker-containe).
[ 1558.892701] systemd-shutdown[1]: Sending SIGKILL to PID 2079 (kolla_start).
[ 1558.897848] systemd-shutdown[1]: Sending SIGKILL to PID 2189 (ceilometer-agen).
[ 1558.903574] systemd-shutdown[1]: Sending SIGKILL to PID 2190 (docker-containe).
[ 1558.909220] systemd-shutdown[1]: Sending SIGKILL to PID 2217 (docker-containe).
[ 1558.914799] systemd-shutdown[1]: Sending SIGKILL to PID 2277 (docker-containe).
[ 1558.920219] systemd-shutdown[1]: Sending SIGKILL to PID 2317 (kolla_start).
[ 1558.924792] systemd-shutdown[1]: Sending SIGKILL to PID 2329 (kolla_start).
[ 1558.932073] systemd-shutdown[1]: Sending SIGKILL to PID 2376 (kolla_start).
[ 1558.943927] systemd-shutdown[1]: Sending SIGKILL to PID 2584 (aodh-notifier: ).
[ 1558.952055] systemd-shutdown[1]: Sending SIGKILL to PID 2588 (docker-containe).
[ 1558.956779] systemd-shutdown[1]: Sending SIGKILL to PID 2612 (cinder-schedule).
[ 1558.961131] systemd-shutdown[1]: Sending SIGKILL to PID 2690 (neutron-l3-agen).

It's worth noting that projects like neutron refined the systemd service behavior over years including the addition of order-dependent cleanup oneshot services like "neutron-ovs-cleanup". These refinements addressed actual issues so some form of graceful shutdown mechanism is desirable to avoid unexpected, difficult to debug, issues.

--- Additional comment from Alex Schultz on 2018-06-27 10:07:11 EDT ---

Yea not sure if this is super configurable from a container runtime  perspective. We'll have to figure this out. That being said, it would be beneficial for the services not to rely on this as other deployment mechanisms with containers might need to rely on different cleanup processes.

Comment 9 Artem Hrechanychenko 2018-08-06 16:12:35 UTC
[heat-admin@controller-2 ~]$ sudo rpm -qa "python-paunch"
python-paunch-2.5.0-2.el7ost.noarch
 

[heat-admin@controller-2 ~]$ journalctl --since "2018-08-06 15:40:00" |grep "Unmounting /var/lib/docker/containers/"



Aug 06 15:52:05 controller-2 systemd[1]: Unmounting /var/lib/docker/containers/39745e25aa13e34b20ead5cae31ebf051371fcc30503a77b2dba41adbeb17149/shm...
Aug 06 15:52:05 controller-2 systemd[1]: Unmounting /var/lib/docker/containers/d04dab3c381ef35ad194d15b1fba1a54bdf021d527cd2748530d78f46675a824/shm...
Aug 06 15:52:05 controller-2 systemd[1]: Unmounting /var/lib/docker/containers/c56bb23f586b4373641ac225d596b16989de96b972f1ff0ffbbbec79eba39803/shm...
Aug 06 15:52:05 controller-2 systemd[1]: Unmounting /var/lib/docker/containers/76dcac8b2af9527bcddb69d880bace272381f16f1d39c10cab4351b1fc9eb07f/shm...
Aug 06 15:52:05 controller-2 systemd[1]: Unmounting /var/lib/docker/containers/db02cab50a7d844598b14561a65a7e59151c7cfb1672e42b19c316fcfa83c074/shm...
Aug 06 15:52:05 controller-2 systemd[1]: Unmounting /var/lib/docker/containers/ccb8cb69a8ba9675c836d2ae92ba5e0c42c8a799147eb91be1c0b4b2aa207f2f/shm...
Aug 06 15:52:05 controller-2 systemd[1]: Unmounting /var/lib/docker/containers/580697e56dc99616e567ed74b866732f7c76b3e230771404e172bdbfb39ef43a/shm...
Aug 06 15:52:05 controller-2 systemd[1]: Unmounting /var/lib/docker/containers/b6b48c390d019a837821e7fd2779300070285b0746256b827204668a89c1048e/shm...
Aug 06 15:52:05 controller-2 systemd[1]: Unmounting /var/lib/docker/containers/7638cae911f0d22d71174e1392042475d10d7d696e2ec06764e63f262e01bb52/shm...
Aug 06 15:52:05 controller-2 systemd[1]: Unmounting /var/lib/docker/containers/e3d6d3f2ce10266f455bb2b0bb8c0459474660015a86986eb998644b9844c610/shm...
Aug 06 15:52:05 controller-2 systemd[1]: Unmounting /var/lib/docker/containers/98b46c1871e75f7e9b68b37bf4f4915137d6e08b00e9826cef854cc059fa9110/shm...
Aug 06 15:52:05 controller-2 systemd[1]: Unmounting /var/lib/docker/containers/29b4a860b226c812a871b52c1cb86738d98af8afe1aeda5e640eeacf70267a76/shm...
Aug 06 15:52:05 controller-2 systemd[1]: Unmounting /var/lib/docker/containers/4e647926317b53333fe2492935d7343d2f11aa2a63e5d6602d1a8082c9318549/shm...
Aug 06 15:52:05 controller-2 systemd[1]: Unmounting /var/lib/docker/containers/a953c4df52a87a357318e33d98bd221ae05b25dbb5f1c4546d485dceedb3955a/shm...
Aug 06 15:52:05 controller-2 systemd[1]: Unmounting /var/lib/docker/containers/64a47b75b377f93adbf15696b78d973d6c1bb628b57d7c21c771e58ef6db0156/shm...
Aug 06 15:52:05 controller-2 systemd[1]: Unmounting /var/lib/docker/containers/cb40c142939e465cd3addaf784436a06b75431cc654b6d65f8c40ead18b2d351/shm...
Aug 06 15:52:05 controller-2 systemd[1]: Unmounting /var/lib/docker/containers/c6589a59c842b46da1a6f8f0b840e86cf2cc76dfd8506866953ce7afd5143f0d/shm...
Aug 06 15:52:05 controller-2 systemd[1]: Unmounting /var/lib/docker/containers/e5cb3e0353927d50d6439e6f60eda30773ae4282a13f057c87166ad973a55312/shm...
Aug 06 15:52:05 controller-2 systemd[1]: Unmounting /var/lib/docker/containers/1e508452468f3a204804df5decb0817d33c5db73ae963f5c899c176cfcc11a58/shm...
Aug 06 15:52:05 controller-2 systemd[1]: Unmounting /var/lib/docker/containers/588aaee553e6dc6f3671ade7702c7b3797a0d6e7609fbd1d767120a6728b3051/shm...
Aug 06 15:52:05 controller-2 systemd[1]: Unmounting /var/lib/docker/containers/ccc6494683da0a19e14a06a2f912fb6815c088cd764f7b073269d5b26d173b2e/shm...
Aug 06 15:52:05 controller-2 systemd[1]: Unmounting /var/lib/docker/containers/85aca71c3f49462d521bb4398c44ad0e03bbf8d67b353b0a16b9f1523b9b984c/shm...
Aug 06 15:52:05 controller-2 systemd[1]: Unmounting /var/lib/docker/containers/3e4da76ec3b5ad311f555fc019fd2279fb5356e567e85579d4e06e80cabbf12d/shm...
Aug 06 15:52:05 controller-2 systemd[1]: Unmounting /var/lib/docker/containers/628f926f43aa8767f5571ceb3a63104453b53c00da4d8566d20081c6d2b35257/shm...
Aug 06 15:52:05 controller-2 systemd[1]: Unmounting /var/lib/docker/containers/246d1137b7222e1fdc4bb2e00128d25069d9d72cb6b2ed147e868013af241b13/shm...
Aug 06 15:52:05 controller-2 systemd[1]: Unmounting /var/lib/docker/containers/91a6dc789e67c9166ab4a2afa0ad591cbc73acc8b285f820a60469c81ae6bf34/shm...
Aug 06 15:52:05 controller-2 systemd[1]: Unmounting /var/lib/docker/containers/21fdbe68bd6bac760579120f32cc6876fef57f1e22ab3c10c9572221b597a86e/shm...
Aug 06 15:52:05 controller-2 systemd[1]: Unmounting /var/lib/docker/containers/390ded04c2d7c044b60566a4a3c06d14ed3c8018690b761fb98b5b21d8316b4e/shm...
Aug 06 15:52:05 controller-2 systemd[1]: Unmounting /var/lib/docker/containers/f42b8119b1e5ac598111d077d7f0f07f4c595da563f9dc663a854f5aface216b/shm...
Aug 06 15:52:05 controller-2 systemd[1]: Unmounting /var/lib/docker/containers/10dbb6673ebdff0f055d2fb3ae204c16e6d8584265db180e85aad33f4a0d3156/shm...
Aug 06 15:52:05 controller-2 systemd[1]: Unmounting /var/lib/docker/containers/8dc8db0b9bd5725a5ac0909284d56e5f9e5ac4b25d3c3c4b013c43feb45fa33c/shm...
Aug 06 15:52:05 controller-2 systemd[1]: Unmounting /var/lib/docker/containers/b9bff738744bb1545870e9efbc05f61220c8e8bc7ff867657eecdd6c64f15b4b/shm...
Aug 06 15:52:05 controller-2 systemd[1]: Unmounting /var/lib/docker/containers/e212d9a58c795142d828a033bcfe0c57c02a49292321a203485db2a7ad0aa966/shm...
Aug 06 15:52:05 controller-2 systemd[1]: Unmounting /var/lib/docker/containers/a87b48af1fa21060697294cbb2668409e98571af8a3d470e9a5f72d812217b4c/shm...
Aug 06 15:52:05 controller-2 systemd[1]: Unmounting /var/lib/docker/containers/9cf91fe2f7b51e1eaf28f0f31d677e213b5f48d8838a9269e84b967ea3369267/shm...
Aug 06 15:52:05 controller-2 systemd[1]: Unmounting /var/lib/docker/containers/f650437e0c76ff01f6c46e5d8e14052fc0f3ddf08c48ba2c04789062c1dd6a0b/shm...
Aug 06 15:52:05 controller-2 systemd[1]: Unmounting /var/lib/docker/containers/895ad96c42739f10dd0f7193375f490e33b256c67c7af9645994a1e3ebcacd46/shm...
Aug 06 15:52:05 controller-2 systemd[1]: Unmounting /var/lib/docker/containers/86af70bab7a28e741c86c81169c9ac74f96727f0cb3d6344060250136e332421/shm...
Aug 06 15:52:05 controller-2 systemd[1]: Unmounting /var/lib/docker/containers/f5f2485ebfda3a3a4e3ced6f1113147de91ffb4564b95b0118f92ad8c40d1afc/shm...
Aug 06 15:52:05 controller-2 systemd[1]: Unmounting /var/lib/docker/containers/a7165d3279c89d654790410e17ff06393ed6be3a03234e3ca138a5493ac08923/shm...
Aug 06 15:52:05 controller-2 systemd[1]: Unmounting /var/lib/docker/containers/df45c478baa1e62882d14d7a5422070ea99c658606f115388c497194fb35c955/shm...
Aug 06 15:52:05 controller-2 systemd[1]: Unmounting /var/lib/docker/containers/e136edde87f7fe3ae73d7e913a9711459051c665320b1c5005d5d281840ee56f/shm...
Aug 06 15:52:05 controller-2 systemd[1]: Unmounting /var/lib/docker/containers/3927cafa0f95ba436ada7dae12a85b2fada2fcce0ff4ff091eebf1898755f429/shm...
Aug 06 15:52:05 controller-2 systemd[1]: Unmounting /var/lib/docker/containers/b54437039c43dd6d258ca9d5beb813697be6ce9bd3d06c5850adba9b3bb5a82a/shm...
Aug 06 15:52:05 controller-2 systemd[1]: Unmounting /var/lib/docker/containers/497ffd81ee40da79361a3aab92e01b5acb4a677776f2723dff3c703cc8d0ad5d/shm...
Aug 06 15:52:05 controller-2 systemd[1]: Unmounting /var/lib/docker/containers/c4b8b1d036402f367ae067353e9b7556e9be574c4724fa5683991eb77f1d5744/shm...
Aug 06 15:52:05 controller-2 systemd[1]: Unmounting /var/lib/docker/containers/98f082ac1504162eebf1e772cee22866f503871b8509e8275302e5e0ca1c7950/shm...
Aug 06 15:52:05 controller-2 systemd[1]: Unmounting /var/lib/docker/containers/66087f57b103df79432cd5c9d272d54590fcefcb17b7d72a28e433e22554b35c/shm...
Aug 06 15:52:05 controller-2 systemd[1]: Unmounting /var/lib/docker/containers/72705ccc232a1e5a9bddaf49c394815275ba749f7a6d926fc6385df8e3ab436a/shm...
Aug 06 15:52:05 controller-2 systemd[1]: Unmounting /var/lib/docker/containers/41cd54f149a199429109712c0812ed1c130a4a6f06668426a1a7a19b9385673b/shm...
Aug 06 15:52:05 controller-2 systemd[1]: Unmounting /var/lib/docker/containers/74afd69dfdb985016ded85e7d98dde6ba9c43993b1136f0469935162fa98f333/shm...
Aug 06 15:52:05 controller-2 systemd[1]: Unmounting /var/lib/docker/containers/42ebb9dde2ed05602b6fea5b909b5d5b6305b3ddfae61d07d72389750503485a/shm...
Aug 06 15:52:05 controller-2 systemd[1]: Unmounting /var/lib/docker/containers/8301553fabce0e76c35cdc9e26b9db1bbf9b5dca73f4c25cba4cea3b232eef02/shm...
Aug 06 15:52:05 controller-2 systemd[1]: Unmounting /var/lib/docker/containers/085a3cd67a69e26fdc58dc6c14a9b31b5e95c0662cadf318572f95118f3a4205/shm...
Aug 06 15:52:05 controller-2 systemd[1]: Unmounting /var/lib/docker/containers/24a380d5342c4772cb0829b9e226552ce3fadaf8c70beb75a01ba6b953d1d42f/shm...



[heat-admin@controller-2 ~]$ journalctl --since "2018-08-06 15:40:00" |grep "(docker-containe)"
[heat-admin@controller-2 ~]$ journalctl --since "2018-08-06 15:40:00" |grep "Sending SIGKILL to PID"
[heat-admin@controller-2 ~]$

Comment 10 Joanne O'Flynn 2018-08-13 11:18:23 UTC
This bug is marked for inclusion in the errata but does not currently contain draft documentation text. To ensure the timely release of this advisory please provide draft documentation text for this bug as soon as possible.

If you do not think this bug requires errata documentation, set the requires_doc_text flag to "-".


To add draft documentation text:

* Select the documentation type from the "Doc Type" drop down field.

* A template will be provided in the "Doc Text" field based on the "Doc Type" value selected. Enter draft text in the "Doc Text" field.

Comment 12 errata-xmlrpc 2018-08-29 16:32:35 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:2573


Note You need to log in before you can comment on or make changes to this bug.