Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 1598191 - overcloud ironic service is missing db sync in fast_forward_upgrade_tasks
Summary: overcloud ironic service is missing db sync in fast_forward_upgrade_tasks
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-tripleo-heat-templates
Version: 13.0 (Queens)
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: z2
: 13.0 (Queens)
Assignee: Dmitry Tantsur
QA Contact: Alexander Chuzhoy
URL:
Whiteboard:
: 1598542 (view as bug list)
Depends On:
Blocks: 1599436
TreeView+ depends on / blocked
 
Reported: 2018-07-04 14:59 UTC by Alexander Chuzhoy
Modified: 2018-12-24 11:40 UTC (History)
7 users (show)

Fixed In Version: openstack-tripleo-heat-templates-8.0.4-3.el7ost
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1599436 (view as bug list)
Environment:
Last Closed: 2018-08-29 16:37:54 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Launchpad 1780116 None None None 2018-07-04 15:38:19 UTC
OpenStack gerrit 580200 None master: MERGED tripleo-heat-templates: FFU: missing ironic tasks (I24d20fad72ce29fd3c2c4a96f1ee8422d3e1bdc3) 2018-07-19 02:14:08 UTC
OpenStack gerrit 581103 None stable/queens: MERGED tripleo-heat-templates: FFU: missing ironic tasks (I24d20fad72ce29fd3c2c4a96f1ee8422d3e1bdc3) 2018-07-19 02:14:01 UTC
RDO 14689 None queens-rdo: MERGED openstack/ironic-distgit: Bump osprofiler dependency to 1.5.0 (Ifd3e3a3433a17bbcc3b7150f091af02637e2b031) 2018-07-19 02:14:14 UTC
Red Hat Product Errata RHBA-2018:2574 None None None 2018-08-29 16:38:52 UTC
Storyboard 2002900 None None None 2018-07-06 14:31:46 UTC

Description Alexander Chuzhoy 2018-07-04 14:59:45 UTC
overcloud ironic service is missing db sync in fast_forward_upgrade_tasks
Environment:
openstack-ironic-conductor-6.2.4-3.el7ost.noarch
puppet-ironic-9.5.0-3.el7ost.noarch
python-ironicclient-1.7.1-1.el7ost.noarch
python-ironic-lib-2.1.3-1.el7ost.noarch
openstack-ironic-common-6.2.4-3.el7ost.noarch
openstack-ironic-api-6.2.4-3.el7ost.noarch
python-ironic-inspector-client-1.10.0-1.el7ost.noarch



During FFU, specifically section 6.2 here:
https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/13/html/fast_forward_upgrades/assembly-upgrading_the_overcloud#upgrading_all_controller_nodes 

the process fails with:
"Error running [\'docker\', \'run\', \'--name\', \'ironic_db_sync\', \'--label\', \'config_id=tripleo_step3\', \'--label\', \'container_name=ironic_db_sync\', \'--label\', \'managed_by=paunch\', \'--label\', \'config_data={\\"start_order\\": 1, \\"image\\": \\"192.168.24.1:8787/rhosp13/openstack-ironic-api:2018-07-03.3\\", \\"command\\": \\"/usr/bin/bootstrap_host_exec ironic_api su ironic -s /bin/bash -c \\\\\'ironic-dbsync --config-file /etc/ironic/ironic.conf\\\\\'\\", \\"user\\": \\"root\\", \\"volumes\\": [\\"/etc/hosts:/etc/hosts:ro\\", \\"/etc/localtime:/etc/localtime:ro\\", \\"/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro\\", \\"/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro\\", \\"/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro\\", \\"/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro\\", \\"/dev/log:/dev/log\\", \\"/etc/ssh/ssh_known_hosts:/etc/ssh/ssh_known_hosts:ro\\", \\"/etc/puppet:/etc/puppet:ro\\", \\"/var/lib/config-data/ironic_api/etc/ironic:/etc/ironic:ro\\", \\"/var/log/containers/ironic:/var/log/ironic\\", \\"/var/log/containers/httpd/ironic-api:/var/log/httpd\\"], \\"net\\": \\"host\\", \\"detach\\": false, \\"privileged\\": false}\', \'--net=host\', \'--privileged=false\', \'--user=root\', \'--volume=/etc/hosts:/etc/hosts:ro\', \'--volume=/etc/localtime:/etc/localtime:ro\', \'--volume=/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro\', \'--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro\', \'--volume=/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro\', \'--volume=/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro\', \'--volume=/dev/log:/dev/log\', \'--volume=/etc/ssh/ssh_known_hosts:/etc/ssh/ssh_known_hosts:ro\', \'--volume=/etc/puppet:/etc/puppet:ro\', \'--volume=/var/lib/config-data/ironic_api/etc/ironic:/etc/ironic:ro\', \'--volume=/var/log/containers/ironic:/var/log/ironic\', \'--volume=/var/log/containers/httpd/ironic-api:/var/log/httpd\', \'192.168.24.1:8787/rhosp13/openstack-ironic-api:2018-07-03.3\', \'/usr/bin/bootstrap_host_exec\', \'ironic_api\', \'su\', \'ironic\', \'-s\', \'/bin/bash\', \'-c\', \\"\'ironic-dbsync\\", \'--config-file\', \\"/etc/ironic/ironic.conf\'\\"]. [1]", ',


Basically,
ironic has no fast forward upgrade tasks for the db sync.

Comment 1 Alexander Chuzhoy 2018-07-05 18:28:35 UTC
*** Bug 1598542 has been marked as a duplicate of this bug. ***

Comment 2 Alexander Chuzhoy 2018-07-05 18:29:47 UTC
With the patch applied, the upgrade fails running the following command:

[root@controller-0 ~]# ironic-dbsync --config-file /etc/ironic/ironic.conf upgrade
Traceback (most recent call last):
  File "/bin/ironic-dbsync", line 6, in <module>
    from ironic.cmd.dbsync import main
  File "/usr/lib/python2.7/site-packages/ironic/cmd/dbsync.py", line 63, in <module>
    dbapi.backfill_version_column,
  File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 243, in __getattr__
    self._load_backend()
  File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 238, in _load_backend
    backend_mod = importutils.import_module(backend_path)
  File "/usr/lib/python2.7/site-packages/oslo_utils/importutils.py", line 73, in import_module
    __import__(import_str)
  File "/usr/lib/python2.7/site-packages/ironic/db/sqlalchemy/api.py", line 203, in <module>
    class Connection(api.Connection):
  File "/usr/lib/python2.7/site-packages/ironic/common/profiler.py", line 59, in decorator
    if CONF.profiler.enabled:
  File "/usr/lib/python2.7/site-packages/oslo_config/cfg.py", line 2489, in __getattr__
    raise NoSuchOptError(name)
oslo_config.cfg.NoSuchOptError: no such option profiler in group [DEFAULT]

Comment 3 Dmitry Tantsur 2018-07-06 09:05:35 UTC
Can you check the versions of all ironic-related packages before and after the upgrade attempt:

 $ rpm -qa | grep ironic

But I suspect that we need to make sure ironic pulls in the right version of osprofiler. Can you check the version of python2-osprofiler before and after upgrade?

What I do not understand though is why you end up running ironic-dbsync outside of the container? The osprofiler support was added in Pike, and IIRC in Pike we should be containerized, no? Maybe this needs input from the upgrades team.

Also please note that I'll be on PTO next week, so we'll need someone to take this over.

Comment 4 Alexander Chuzhoy 2018-07-06 13:46:07 UTC
Tried manually to run update between osp11 and osp12 (this is where the problem starts).
Checking yum history:

Transaction performed with:
    Installed     rpm-4.11.3-32.el7.x86_64                    @anaconda/7.5
    Installed     subscription-manager-1.20.11-1.el7_5.x86_64 @koji-override-0/7.5
    Installed     yum-3.4.3-158.el7.noarch                    @anaconda/7.5
    Installed     yum-metadata-parser-1.1.4-10.el7.x86_64     @anaconda/7.5
Packages Altered:
    Updated     openstack-ironic-api-1:7.0.4-1.el7ost.noarch       @rhelosp-11.0-puddle
    Update                           1:9.1.4-1.el7ost.noarch       @rhelosp-12.0-puddle
    Updated     openstack-ironic-common-1:7.0.4-1.el7ost.noarch    @rhelosp-11.0-puddle
    Update                              1:9.1.4-1.el7ost.noarch    @rhelosp-12.0-puddle
    Updated     openstack-ironic-conductor-1:7.0.4-1.el7ost.noarch @rhelosp-11.0-puddle
    Update                                 1:9.1.4-1.el7ost.noarch @rhelosp-12.0-puddle
    Updated     python-amqp-1.4.6-1.el7ost.noarch                  @rhos-10.0-signed
    Update                  2.1.4-2.el7ost.noarch                  @rhelosp-12.0-puddle
    Updated     python-cinderclient-1.9.0-6.el7ost.noarch          @rhos-10.0-signed
    Update                          3.1.0-1.el7ost.noarch          @rhelosp-12.0-puddle
    Updated     python-glanceclient-1:2.5.0-2.el7ost.noarch        @rhos-10.0-signed
    Update                          1:2.8.0-2.el7ost.noarch        @rhelosp-12.0-puddle
    Updated     python-kombu-1:3.0.32-2.el7ost.noarch              @rhos-10.0-signed
    Update                   1:4.0.2-5.el7ost.noarch               @rhelosp-12.0-puddle
    Updated     python-migrate-0.10.0-1.el7ost.noarch              @rhos-10.0-signed
    Update                     0.11.0-1.el7ost.noarch              @rhelosp-12.0-puddle
    Updated     python-neutronclient-6.0.1-1.el7ost.noarch         @rhos-10.0-signed
    Update                           6.5.0-1.el7ost.noarch         @rhelosp-12.0-puddle
    Updated     python-oslo-db-4.17.1-1.el7ost.noarch              @rhelosp-11.0-puddle
    Update                     4.25.1-1.el7ost.noarch              @rhelosp-12.0-puddle
    Updated     python-oslo-db-lang-4.17.1-1.el7ost.noarch         @rhelosp-11.0-puddle
    Update                          4.25.1-1.el7ost.noarch         @rhelosp-12.0-puddle
    Updated     python-oslo-messaging-5.17.3-1.el7ost.noarch       @rhelosp-11.0-puddle
    Update                            5.30.2-1.el7ost.noarch       @rhelosp-12.0-puddle
    Updated     python-oslo-middleware-3.19.1-1.el7ost.noarch      @rhos-10.0-signed
    Update                             3.30.1-1.el7ost.noarch      @rhelosp-12.0-puddle
    Updated     python-oslo-middleware-lang-3.19.1-1.el7ost.noarch @rhos-10.0-signed
    Update                                  3.30.1-1.el7ost.noarch @rhelosp-12.0-puddle
    Updated     python-oslo-policy-1.18.0-1.el7ost.noarch          @rhelosp-11.0-puddle
    Update                         1.25.2-1.el7ost.noarch          @rhelosp-12.0-puddle
    Updated     python-oslo-policy-lang-1.18.0-1.el7ost.noarch     @rhelosp-11.0-puddle
    Update                              1.25.2-1.el7ost.noarch     @rhelosp-12.0-puddle
    Updated     python-proliantutils-2.2.0-3.el7ost.noarch         @rhos-10.0-signed
    Update                           2.4.0-4.el7ost.noarch         @rhelosp-12.0-puddle
    Dep-Install python-scciclient-0.5.0-1.el7ost.noarch            @rhelosp-12.0-puddle
    Dep-Install python-sushy-1.1.0-1.el7ost.noarch                 @rhelosp-12.0-puddle
    Dep-Install python-vine-1.1.3-2.el7ost.noarch                  @rhelosp-12.0-puddle
    Updated     python-webob-1.6.1-2.el7ost.noarch                 @rhos-10.0-signed
    Update                   1.7.2-1.el7ost.noarch                 @rhelosp-12.0-puddle
    Dep-Install python2-statsd-3.2.1-5.el7ost.noarch               @rhelosp-11.0-puddle





Packages Altered:
    Updated    python-oslo-cache-1.14.0-1.1.el7ost.noarch               @rhos-10.0-signed
    Update                       1.25.1-1.el7ost.noarch                 @rhelosp-12.0-puddle
    Updated    python-oslo-cache-lang-1.14.0-1.1.el7ost.noarch          @rhos-10.0-signed
    Update                            1.25.1-1.el7ost.noarch            @rhelosp-12.0-puddle
    Updated    python-oslo-concurrency-3.14.1-1.el7ost.noarch           @rhos-10.0-signed
    Update                             3.21.1-1.el7ost.noarch           @rhelosp-12.0-puddle
    Updated    python-oslo-concurrency-lang-3.14.1-1.el7ost.noarch      @rhos-10.0-signed
    Update                                  3.21.1-1.el7ost.noarch      @rhelosp-12.0-puddle
    Updated    python-oslo-privsep-1.13.2-1.el7ost.noarch               @rhos-10.0-signed
    Update                         1.22.1-1.el7ost.noarch               @rhelosp-12.0-puddle
    Updated    python-oslo-privsep-lang-1.13.2-1.el7ost.noarch          @rhos-10.0-signed
    Update                              1.22.1-1.el7ost.noarch          @rhelosp-12.0-puddle
    Updated    python-oslo-reports-1.14.0-1.el7ost.noarch               @rhos-10.0-signed
    Update                         1.22.1-1.el7ost.noarch               @rhelosp-12.0-puddle
    Updated    python-oslo-rootwrap-5.1.2-1.el7ost.noarch               @rhos-10.0-signed
    Update                          5.9.2-1.el7ost.noarch               @rhelosp-12.0-puddle
    Updated    python-oslo-service-1.16.1-1.el7ost.noarch               @rhos-10.0-signed
    Update                         1.25.1-1.el7ost.noarch               @rhelosp-12.0-puddle
    Updated    python-oslo-sphinx-4.7.0-1.el7ost.noarch                 @rhos-10.0-signed
    Obsoleted  python-oslo-sphinx-4.7.0-1.el7ost.noarch                 @rhos-10.0-signed
    Obsoleting python-oslo-sphinx-4.15.1-2.el7ost.noarch                @rhelosp-12.0-puddle
    Updated    python-oslo-versionedobjects-1.17.1-1.el7ost.noarch      @rhos-10.0-signed
    Update                                  1.26.2-1.el7ost.noarch      @rhelosp-12.0-puddle
    Updated    python-oslo-versionedobjects-lang-1.17.1-1.el7ost.noarch @rhos-10.0-signed
    Update                                       1.26.2-1.el7ost.noarch @rhelosp-12.0-puddle
    Updated    python-oslo-vmware-2.14.1-1.el7ost.noarch                @rhos-10.0-signed
    Update                        2.23.1-1.el7ost.noarch                @rhelosp-12.0-puddle
    Updated    python-oslo-vmware-lang-2.14.1-1.el7ost.noarch           @rhos-10.0-signed
    Update                             2.23.1-1.el7ost.noarch           @rhelosp-12.0-puddle
    Updated    python-oslotest-2.10.0-1.el7ost.noarch                   @rhos-10.0-signed
    Update                     2.17.1-1.el7ost.noarch                   @rhelosp-12.0-puddle

Comment 5 Alexander Chuzhoy 2018-07-06 13:56:12 UTC
If python-osprofiler is updated to a version from osp12 - the ironic-dbsync doesn't show the error:

Dependencies Resolved

==============================================================================================================================================================================================
 Package                                         Arch                                 Version                                         Repository                                         Size
==============================================================================================================================================================================================
Updating:
 python-osprofiler                               noarch                               1.11.0-1.el7ost                                 rhelosp-12.0-puddle                               115 k

Transaction Summary
==============================================================================================================================================================================================
Upgrade  1 Package

Total download size: 115 k
Is this ok [y/d/N]: y
Downloading packages:
No Presto metadata available for rhelosp-12.0-puddle
python-osprofiler-1.11.0-1.el7ost.noarch.rpm                                                                                                                           | 115 kB  00:00:00     
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Updating   : python-osprofiler-1.11.0-1.el7ost.noarch                                                                                                                                   1/2 
  Cleanup    : python-osprofiler-1.4.0-1.el7ost.noarch                                                                                                                                    2/2 
  Verifying  : python-osprofiler-1.11.0-1.el7ost.noarch                                                                                                                                   1/2 
  Verifying  : python-osprofiler-1.4.0-1.el7ost.noarch                                                                                                                                    2/2 

Updated:
  python-osprofiler.noarch 0:1.11.0-1.el7ost                                                                                                                                                  

Complete!
[root@controller-1 ~]# ironic-dbsync --config-file /etc/ironic/ironic.conf upgrade
Option "rpc_backend" from group "DEFAULT" is deprecated for removal (Replaced by [DEFAULT]/transport_url).  Its value may be silently ignored in the future.
INFO  [alembic.runtime.migration] Context impl MySQLImpl.
INFO  [alembic.runtime.migration] Will assume non-transactional DDL.
INFO  [alembic.runtime.migration] Running upgrade dbefd6bdaa2c -> 3d86a077a3f2, add port physical network
INFO  [alembic.runtime.migration] Running upgrade 3d86a077a3f2 -> 868cb606a74a, add version field in base class
[root@controller-1 ~]# echo $?
0

Comment 16 Joanne O'Flynn 2018-08-15 07:56:43 UTC
This bug is marked for inclusion in the errata but does not currently contain draft documentation text. To ensure the timely release of this advisory please provide draft documentation text for this bug as soon as possible.

If you do not think this bug requires errata documentation, set the requires_doc_text flag to "-".


To add draft documentation text:

* Select the documentation type from the "Doc Type" drop down field.

* A template will be provided in the "Doc Text" field based on the "Doc Type" value selected. Enter draft text in the "Doc Text" field.

Comment 17 Alexander Chuzhoy 2018-08-21 20:08:22 UTC
Environment:
openstack-tripleo-heat-templates-8.0.4-20.el7ost.noarch


The reported issue doesn't reproduce.

Comment 19 errata-xmlrpc 2018-08-29 16:37:54 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:2574


Note You need to log in before you can comment on or make changes to this bug.