Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.

Bug 1597100

Summary: failing to import and register the nodes to the director, getting error Background on this error at: http://sqlalche.me/e/f405)
Product: Red Hat OpenStack Reporter: svegesna <sudheer.vegesna>
Component: openstack-zaqarAssignee: Thomas Hervé <therve>
Status: CLOSED NOTABUG QA Contact: Shai Revivo <srevivo>
Severity: urgent Docs Contact:
Priority: medium    
Version: 13.0 (Queens)CC: anand.rathi, apevec, bfournie, carl.tung, hemal.shah, jschluet, lhh, mburns, rrasouli, srevivo, sriharsha.basavapatna, sudheer.vegesna, therve
Target Milestone: ---Keywords: Tracking, Triaged
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-07-25 11:59:07 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Attachments:
Description Flags
install-undercloud none

Description svegesna 2018-07-02 04:01:30 UTC
OS: RHEL 7.5 GA.

trying with minimum requirements 

1 host machine for the Red Hat OpenStack Platform director(installed RHEL 7.5 GA on this machine)
1 host machine for a Red Hat OpenStack Platform Compute node
1 host machine for a Red Hat OpenStack Platform Controller node

Steps to Reproduce:
1.
(undercloud) [sudheer@facebook ~]$ openstack overcloud node import instackenv.json

{u'body': {u'exception': u'(pymysql.err.ProgrammingError) (1146, u"Table \'zaqar.Queues\' doesn\'t exist") [SQL: u\'INSERT INTO `Queues` (project, name, metadata) VALUES (%(project)s, %(name)s, %(metadata)s)\'] [parameters: {\'project\': u\'13835fbb8e0947a9b3fa174b9a22cdb9\', \'name\': u\'tripleo\', \'metadata\': bytearray(b\'{}\')}] (Background on this error at: http://sqlalche.me/e/f405)', u'error': u'Unexpected error.'}, u'headers': {u'status': 500}, u'request': {u'action': u'queue_create', u'body': {u'queue_name': u'tripleo'}, u'api': u'v2', u'headers': {u'Client-ID': u'c942a1d5-e97d-4d41-8c17-ceba8e26db93', u'X-Project-ID': u'13835fbb8e0947a9b3fa174b9a22cdb9'}}}


Additional info:


(undercloud) [sudheer@facebook ~]$ openstack image list
+--------------------------------------+------------------------+--------+
| ID                                   | Name                   | Status |
+--------------------------------------+------------------------+--------+
| c80dfdd2-68c4-416d-b27e-4cb6c1d20ca4 | bm-deploy-kernel       | active |
| 1063d0c9-0748-48af-8275-701a00c8dfc6 | bm-deploy-ramdisk      | active |
| 5c21f9a7-7972-4f2c-aa18-ee3d2297f29e | overcloud-full         | active |
| d7f05bc6-471b-4c92-a79f-5951b1eb3535 | overcloud-full-initrd  | active |
| 01b24dfa-cbe7-4b25-a940-225ecf51d400 | overcloud-full-vmlinuz | active |
+--------------------------------------+------------------------+--------+

(undercloud) [sudheer@facebook ~]$ cat instackenv.json
{
"nodes":[
    {
        "mac":[
            "00:30:48:CF:75:2A"
        ],
        "cpu":"4",
        "memory":"6144",
        "disk":"40",
        "arch":"x86_64",
        "name":"control",
        "pm_type":"pxe_ipmitool",
        "pm_user":"ADMIN",
        "pm_password":"ADMIN",
        "pm_addr":"10.123.155.3"
    },
    {
        "mac":[
            "00:30:48:CF:75:2B"
        ],
        "cpu":"4",
        "memory":"6144",
        "disk":"40",
        "arch":"x86_64",
        "name":"compute",
        "pm_type":"pxe_ipmitool",
        "pm_user":"ADMIN",
        "pm_password":"ADMIN",
        "pm_addr":"10.123.154.185"
    }
]
}
(undercloud) [sudheer@facebook ~]$

Comment 1 Bob Fournier 2018-07-02 13:37:56 UTC
Did the undercloud install properly and can you provide the undercloud install logs?

Is the permissions for the non-root user set up according to https://docs.openstack.org/tripleo-docs/latest/install/installation/installation.html#installing-the-undercloud ?

Comment 2 svegesna 2018-07-02 14:33:40 UTC
yes the undercloud install is successfully installed, attaching the undercloud install log, all the permissions for the non-root users are set.

Comment 3 svegesna 2018-07-02 14:47:08 UTC
Created attachment 1455969 [details]
install-undercloud

Comment 4 Bob Fournier 2018-07-03 18:36:48 UTC
Thanks. Yeah it looks like the undercloud install went fine after 3 failures.  The final set of data we'd need is the log files from /var/log, especially /var/log/messages and /var/log/mistral/engine.log+executor.log but all would be useful. Can you tar these or run sosreport?

Comment 5 svegesna 2018-07-04 05:38:14 UTC
Hi Bob, 

the sosreprot is a 38MB file, so i am not able to attach.

please let me know if you can download it from the below link.

https://ent.box.com/s/2vleeorhuux6fsd9uxujvp5lv5ax3nuw.

Thanks,
Sudheer.

Comment 6 Bob Fournier 2018-07-04 10:41:13 UTC
Thanks Sudheer. Yes I can download it from there.

Comment 7 Bob Fournier 2018-07-05 17:11:27 UTC
From sosreport, zaqar.log is full of these error messages:

2018-07-01 04:32:02.478 5101 ERROR zaqar.transport.wsgi.v2_0.claims [req-3fbb8af2-e3ce-44ff-8199-a31ac41be1c6 6e71dffd643e4c24a0efff2673fdac32 13835fbb8e0947a9b3fa174b9a22cdb9 - default default] (pymysql.err.ProgrammingError) (1146, u"Table 'zaqar.Queues' doesn't exist") [SQL: u'SELECT `Queues`.metadata \nFROM `Queues` \nWHERE `Queues`.project = %(project_1)s AND `Queues`.name = %(name_1)s'] [parameters: {u'name_1': 'tripleo-ui-logging', u'project_1': u'13835fbb8e0947a9b3fa174b9a22cdb9'}] (Background on this error at: http://sqlalche.me/e/f405): ProgrammingError: (pymysql.err.ProgrammingError) (1146, u"Table 'zaqar.Queues' doesn't exist") [SQL: u'SELECT `Queues`.metadata \nFROM `Queues` \nWHERE `Queues`.project = %(project_1)s AND `Queues`.name = %(name_1)s'] [parameters: {u'name_1': 'tripleo-ui-logging', u'project_1': u'13835fbb8e0947a9b3fa174b9a22cdb9'}] (Background on this error at: http://sqlalche.me/e/f405)
2018-07-01 04:32:02.478 5101 ERROR zaqar.transport.wsgi.v2_0.claims   File "/usr/lib/python2.7/site-packages/zaqar/transport/wsgi/v2_0/claims.py", line 85, in on_post
2018-07-01 04:32:02.478 5101 ERROR zaqar.transport.wsgi.v2_0.claims     **claim_options)
2018-07-01 04:32:02.478 5101 ERROR zaqar.transport.wsgi.v2_0.claims   File "/usr/lib/python2.7/site-packages/zaqar/common/pipeline.py", line 97, in consumer
2018-07-01 04:32:02.478 5101 ERROR zaqar.transport.wsgi.v2_0.claims     tmp = target(*args, **kwargs)
2018-07-01 04:32:02.478 5101 ERROR zaqar.transport.wsgi.v2_0.claims   File "/usr/lib/python2.7/site-packages/zaqar/storage/swift/claims.py", line 98, in create
2018-07-01 04:32:02.478 5101 ERROR zaqar.transport.wsgi.v2_0.claims     queue_meta = queue_ctrl.get_metadata(queue, project=project)
2018-07-01 04:32:02.478 5101 ERROR zaqar.transport.wsgi.v2_0.claims   File "/usr/lib/python2.7/site-packages/zaqar/storage/sqlalchemy/queues.py", line 67, in get_metadata
2018-07-01 04:32:02.478 5101 ERROR zaqar.transport.wsgi.v2_0.claims     queue = self.driver.run(sel).fetchone()
2018-07-01 04:32:02.478 5101 ERROR zaqar.transport.wsgi.v2_0.claims   File "/usr/lib/python2.7/site-packages/zaqar/storage/sqlalchemy/driver.py", line 66, in run
2018-07-01 04:32:02.478 5101 ERROR zaqar.transport.wsgi.v2_0.claims     return self.engine.execute(*args, **kwargs)
2018-07-01 04:32:02.478 5101 ERROR zaqar.transport.wsgi.v2_0.claims   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2075, in execute
2018-07-01 04:32:02.478 5101 ERROR zaqar.transport.wsgi.v2_0.claims     return connection.execute(statement, *multiparams, **params)
2018-07-01 04:32:02.478 5101 ERROR zaqar.transport.wsgi.v2_0.claims   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 948, in execute
2018-07-01 04:32:02.478 5101 ERROR zaqar.transport.wsgi.v2_0.claims     return meth(self, multiparams, params)
2018-07-01 04:32:02.478 5101 ERROR zaqar.transport.wsgi.v2_0.claims   File "/usr/lib64/python2.7/site-packages/sqlalchemy/sql/elements.py", line 269, in _execute_on_connection
2018-07-01 04:32:02.478 5101 ERROR zaqar.transport.wsgi.v2_0.claims     return connection._execute_clauseelement(self, multiparams, params)
2018-07-01 04:32:02.478 5101 ERROR zaqar.transport.wsgi.v2_0.claims   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1060, in _execute_clauseelement
2018-07-01 04:32:02.478 5101 ERROR zaqar.transport.wsgi.v2_0.claims     compiled_sql, distilled_params
2018-07-01 04:32:02.478 5101 ERROR zaqar.transport.wsgi.v2_0.claims   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1200, in _execute_context
2018-07-01 04:32:02.478 5101 ERROR zaqar.transport.wsgi.v2_0.claims     context)
2018-07-01 04:32:02.478 5101 ERROR zaqar.transport.wsgi.v2_0.claims   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1409, in _handle_dbapi_exception
2018-07-01 04:32:02.478 5101 ERROR zaqar.transport.wsgi.v2_0.claims     util.raise_from_cause(newraise, exc_info)
2018-07-01 04:32:02.478 5101 ERROR zaqar.transport.wsgi.v2_0.claims   File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause
2018-07-01 04:32:02.478 5101 ERROR zaqar.transport.wsgi.v2_0.claims     reraise(type(exception), exception, tb=exc_tb, cause=cause)
2018-07-01 04:32:02.478 5101 ERROR zaqar.transport.wsgi.v2_0.claims   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1193, in _execute_context
2018-07-01 04:32:02.478 5101 ERROR zaqar.transport.wsgi.v2_0.claims     context)
2018-07-01 04:32:02.478 5101 ERROR zaqar.transport.wsgi.v2_0.claims   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 507, in do_execute
2018-07-01 04:32:02.478 5101 ERROR zaqar.transport.wsgi.v2_0.claims     cursor.execute(statement, parameters)
2018-07-01 04:32:02.478 5101 ERROR zaqar.transport.wsgi.v2_0.claims   File "/usr/lib/python2.7/site-packages/pymysql/cursors.py", line 166, in execute
2018-07-01 04:32:02.478 5101 ERROR zaqar.transport.wsgi.v2_0.claims     result = self._query(query)
2018-07-01 04:32:02.478 5101 ERROR zaqar.transport.wsgi.v2_0.claims   File "/usr/lib/python2.7/site-packages/pymysql/cursors.py", line 322, in _query
2018-07-01 04:32:02.478 5101 ERROR zaqar.transport.wsgi.v2_0.claims     conn.query(q)
2018-07-01 04:32:02.478 5101 ERROR zaqar.transport.wsgi.v2_0.claims   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 856, in query
2018-07-01 04:32:02.478 5101 ERROR zaqar.transport.wsgi.v2_0.claims     self._affected_rows = self._read_query_result(unbuffered=unbuffered)
2018-07-01 04:32:02.478 5101 ERROR zaqar.transport.wsgi.v2_0.claims   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 1057, in _read_query_result
2018-07-01 04:32:02.478 5101 ERROR zaqar.transport.wsgi.v2_0.claims     result.read()
2018-07-01 04:32:02.478 5101 ERROR zaqar.transport.wsgi.v2_0.claims   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 1340, in read
2018-07-01 04:32:02.478 5101 ERROR zaqar.transport.wsgi.v2_0.claims     first_packet = self.connection._read_packet()
2018-07-01 04:32:02.478 5101 ERROR zaqar.transport.wsgi.v2_0.claims   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 1014, in _read_packet
2018-07-01 04:32:02.478 5101 ERROR zaqar.transport.wsgi.v2_0.claims     packet.check_error()
2018-07-01 04:32:02.478 5101 ERROR zaqar.transport.wsgi.v2_0.claims   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 393, in check_error
2018-07-01 04:32:02.478 5101 ERROR zaqar.transport.wsgi.v2_0.claims     err.raise_mysql_exception(self._data)
2018-07-01 04:32:02.478 5101 ERROR zaqar.transport.wsgi.v2_0.claims   File "/usr/lib/python2.7/site-packages/pymysql/err.py", line 107, in raise_mysql_exception
2018-07-01 04:32:02.478 5101 ERROR zaqar.transport.wsgi.v2_0.claims     raise errorclass(errno, errval)
2018-07-01 04:32:02.478 5101 ERROR zaqar.transport.wsgi.v2_0.claims ProgrammingError: (pymysql.err.ProgrammingError) (1146, u"Table 'zaqar.Queues' doesn't exist") [SQL: u'SELECT `Queues`.metadata \nFROM `Queues` \nWHERE `Queues`.project = %(project_1)s AND `Queues`.name = %(name_1)s'] [parameters: {u'name_1': 'tripleo-ui-logging', u'project_1': u'13835fbb8e0947a9b3fa174b9a22cdb9'}] (Background on this error at: http://sqlalche.me/e/f405)
2018-07-01 04:32:02.478 5101 ERROR zaqar.transport.wsgi.v2_0.claims

This looks similar to https://bugzilla.redhat.com/show_bug.cgi?id=1464247, however that was fixed in OSP-12 with openstack-zaqar-5.0.1-0.20170905222047.el7ost.

On this system, 6.0.1 is installed.
Installed: 1:openstack-zaqar-6.0.1-1.el7ost.noarch

Sending this to the CloudApp team for further analysis.

Comment 8 Thomas Hervé 2018-07-16 08:15:16 UTC
This looks like a repository issue:

Error: /Stage[main]/Neutron/Package[neutron]/ensure: change from purged to present failed: Execution of '/bin/yum -d 0 -e 0 -y install openstack-neutron' returned 1: Error downloading packages:
2018-06-26 09:25:58,311 INFO:   dnsmasq-utils-2.76-2.el7_4.2.x86_64: [Errno 256] No more mirrors to try.
2018-06-26 09:25:58,311 INFO:   python-ryu-4.15-1.el7ost.noarch: [Errno 256] No more mirrors to try.
2018-06-26 09:25:58,312 INFO:   python-ryu-common-4.15-1.el7ost.noarch: [Errno 256] No more mirrors to try.

Later on we see that with zaqar-db-sync:

Warning: /Stage[main]/Zaqar::Db::Sync/Exec[zaqar-db-sync]: Skipping because of failed dependencies

I don't why it didn't fail, but the install didn't work.

Comment 9 svegesna 2018-07-16 09:31:30 UTC
In the "install-undercloud" i can see this error, after this error the installation failed, after 3 failures, the installation completed.

in the successful case i am not seeing the error.

now i manually tied the command.

sudo yum -y install openstack-neutron
Loaded plugins: langpacks, product-id, search-disabled-repos, subscription-manager
Nothing to do


Thanks,
Sudheer.

Comment 10 Thomas Hervé 2018-07-16 09:46:47 UTC
The issue is that the zaqar-db-sync didn't run. I don't know which run succeeded, I don't have the logs of that one? The one which is linked failed at least. Can you provide the logs again? Or maybe reinstall the undercloud.

Comment 11 svegesna 2018-07-25 10:21:05 UTC
Finally on a completely new setup starting from the scarp i am able to run the command "openstack overcloud node import ./instackenv.json" successfully

(undercloud) [stack@deloite images]$ openstack overcloud node import ./instackenv.json
Started Mistral Workflow tripleo.baremetal.v1.register_or_update. Execution ID: e0c9018e-9437-40a8-8385-c81b3c39aad7
Waiting for messages on queue 'tripleo' with no timeout.


2 node(s) successfully moved to the "manageable" state.
Successfully registered node UUID 90d16eb7-d1ae-4970-bdfa-1c84b1117b37
Successfully registered node UUID 812918b4-716d-47b4-8f6b-9993107e274c
 

so will continue with the reaming steps to setup the OSP13 on the system.

Thanks,
Sudheer.

Comment 12 Thomas Hervé 2018-07-25 11:59:07 UTC
Thanks. Closing this one for now, don't hesitate to reopen if the issue persists.