Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 1695541 - [OSP15][Overcloud] Overcloud deployment failed on starting container-puppet-rabbit_init_tasks container - Failed to apply catalog: Cannot parse invalid user line
Summary: [OSP15][Overcloud] Overcloud deployment failed on starting container-puppet-r...
Keywords:
Status: POST
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-tripleo-heat-templates
Version: 15.0 (Stein)
Hardware: All
OS: Linux
high
high
Target Milestone: ga
: ---
Assignee: Michele Baldessari
QA Contact: Sasha Smolyak
URL:
Whiteboard:
: 1693855 (view as bug list)
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-04-03 10:14 UTC by Artem Hrechanychenko
Modified: 2019-04-11 11:18 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Launchpad 1822673 None None None 2019-04-03 11:44:18 UTC
OpenStack gerrit 649696 None None None 2019-04-04 10:55:38 UTC
OpenStack gerrit 649697 None None None 2019-04-04 10:55:38 UTC
RDO 19963 None None None 2019-04-05 07:42:52 UTC

Description Artem Hrechanychenko 2019-04-03 10:14:31 UTC
Description of problem:
OSP15 Compose - RHOS_TRUNK-15.0-RHEL-8-20190402.n.0 

3controller+1compute


"2019-04-03 09:17:03,843 ERROR: 44061 -- ['/usr/bin/podman', 'run', '--user', 'root', '--name', 'container-puppet-rabbit_in
it_tasks', '--env', 'PUPPET_TAGS=file,file_line,concat,augeas,cron,rabbitmq_policy,rabbitmq_user', '--env', 'NAME=rabbit_init_tasks
', '--env', 'HOSTNAME=controller-0', '--env', 'NO_ARCHIVE=true', '--env', 'STEP=2', '--env', 'NET_HOST=true', '--log-driver', 'json
-file', '--volume', '/etc/localtime:/etc/localtime:ro', '--volume', '/tmp/tmpk_m20olk:/etc/config.pp:ro', '--volume', '/etc/puppet/
:/tmp/puppet-etc/:ro', '--volume', '/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro', '--volume', '/etc/pki/tls/certs/ca
-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro', '--volume', '/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.
trust.crt:ro', '--volume', '/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro', '--volume', '/var/lib/config-data:/var/lib/config-data
/:rw', '--volume', '/dev/log:/dev/log:rw', '--log-opt', 'path=/var/log/containers/stdouts/container-puppet-rabbit_init_tasks.log', 
'--security-opt', 'label=disable', '--volume', '/usr/share/openstack-puppet/modules/:/usr/share/openstack-puppet/modules/:ro', '--v
olume', '/var/lib/config-data/rabbitmq/etc/rabbitmq/:/etc/rabbitmq/:ro', '--volume', '/var/lib/rabbitmq:/var/lib/rabbitmq:z', '--en
trypoint', '/var/lib/container-puppet/container-puppet.sh', '--net', 'host', '--volume', '/etc/hosts:/etc/hosts:ro', '--volume', '/
var/lib/container-puppet/container-puppet.sh:/var/lib/container-puppet/container-puppet.sh:ro', '192.168.24.1:8787/rhosp15/openstac
k-rabbitmq:20190402.1'] run failed after + mkdir -p /etc/puppet",
        "+ '[' -n file,file_line,concat,augeas,cron,rabbitmq_policy,rabbitmq_user ']'",
        "+ TAGS='--tags file,file_line,concat,augeas,cron,rabbitmq_policy,rabbitmq_user'",
        "+ origin_of_time=/var/lib/config-data/rabbit_init_tasks.origin_of_time",
        "+ touch /var/lib/config-data/rabbit_init_tasks.origin_of_time",
        "+ /usr/bin/puppet apply --summarize --detailed-exitcodes --color=false --logdest syslog --logdest console --modulepath=/et
c/puppet/modules:/usr/share/openstack-puppet/modules --tags file,file_line,concat,augeas,cron,rabbitmq_policy,rabbitmq_user /etc/co
nfig.pp",
        "Error: /Stage[main]/Tripleo::Profile::Base::Rabbitmq/Rabbitmq_policy[ha-all@/]: Could not evaluate: cannot parse line from
 list_policies:warning: the VM is running with native name encoding of latin1 which may cause Elixir to malfunction as it expects u
tf8. Please ensure your locale is set to UTF-8 (which can be verified by running \"locale\" in your shell)",
        "Error: Failed to apply catalog: Cannot parse invalid user line: warning: the VM is running with native name encoding of la
tin1 which may cause Elixir to malfunction as it expects utf8. Please ensure your locale is set to UTF-8 (which can be verified by 
running \"locale\" in your shell)",
        "+ rc=1",
        "+ '[' 1 -ne 2 -a 1 -ne 0 ']'",
        "+ exit 1",
        " attempt(s): 1",
        "2019-04-03 09:17:03,843 WARNING: 44061 -- Retrying running container: rabbit_init_tasks",
        "2019-04-03 09:17:29,471 ERROR: 44061 -- ['/usr/bin/podman', 'start', '-a', 'container-puppet-rabbit_init_tasks'] run failed after + mkdir -p /etc/puppet",
        " attempt(s): 2",
        "2019-04-03 09:17:29,472 WARNING: 44061 -- Retrying running container: rabbit_init_tasks",
        "2019-04-03 09:17:54,019 ERROR: 44061 -- ['/usr/bin/podman', 'start', '-a', 'container-puppet-rabbit_init_tasks'] run failed after + mkdir -p /etc/puppet",
        " attempt(s): 3",
        "2019-04-03 09:17:54,019 WARNING: 44061 -- Retrying running container: rabbit_init_tasks",
 "2019-04-03 09:17:54,019 WARNING: 44061 -- Retrying running container: rabbit_init_tasks",
        "2019-04-03 09:17:54,019 ERROR: 44061 -- Failed running container for rabbit_init_tasks",
        "2019-04-03 09:17:54,019 INFO: 44061 -- Finished processing puppet configs for rabbit_init_tasks",
        "2019-04-03 09:17:54,020 ERROR: 44059 -- ERROR configuring rabbit_init_tasks"
    ]
}
skipping: [controller-1] => {}
skipping: [controller-2] => {}
skipping: [compute-0] => {}

NO MORE HOSTS LEFT *************************************************************

PLAY RECAP *********************************************************************
compute-0                  : ok=154  changed=74   unreachable=0    failed=0   
controller-0               : ok=216  changed=127  unreachable=0    failed=1   
controller-1               : ok=213  changed=126  unreachable=0    failed=0   
controller-2               : ok=213  changed=126  unreachable=0    failed=0   
undercloud                 : ok=11   changed=8    unreachable=0    failed=0   

Wednesday 03 April 2019  05:17:54 -0400 (0:00:00.191)       0:21:08.542 ******* 
=============================================================================== 
Exception occured while running the command
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/tripleoclient/command.py", line 30, in run
    super(Command, self).run(parsed_args)
  File "/usr/lib/python3.6/site-packages/osc_lib/command/command.py", line 41, in run
    return super(Command, self).run(parsed_args)
  File "/usr/lib/python3.6/site-packages/cliff/command.py", line 184, in run
    return_code = self.take_action(parsed_args) or 0
  File "/usr/lib/python3.6/site-packages/tripleoclient/v1/overcloud_deploy.py", line 949, in take_action
    verbosity=self.app_args.verbose_level)
  File "/usr/lib/python3.6/site-packages/tripleoclient/workflows/deployment.py", line 327, in config_download
    raise exceptions.DeploymentError("Overcloud configuration failed.")
tripleoclient.exceptions.DeploymentError: Overcloud configuration failed.
Overcloud configuration failed.

Ansible failed, check log at /var/lib/mistral/overcloud/ansible.log.



controller-0:

21f749fc3a80  192.168.24.1:8787/rhosp15/openstack-rabbitmq:20190402.1            /var/lib/containe...  About an hour ago  Exited (1) About an hour ago         container-puppet-rabbit_init_tasks

+ /usr/bin/puppet apply --summarize --detailed-exitcodes --color=false --logdest syslog --logdest console --modulepath=/etc/puppet/modules:/usr/share/openstack-puppet/modules --tags file,file_line,concat,augeas,cron,rabbitmq_policy,rabbitmq_user /etc/config.pp
Warning: /etc/puppet/hiera.yaml: Use of 'hiera.yaml' version 3 is deprecated. It should be converted to version 5
   (file: /etc/puppet/hiera.yaml)
Warning: Undefined variable '::deploy_config_name'; \n   (file & line not available)
Warning: The function 'hiera' is deprecated in favor of using 'lookup'. See https://puppet.com/docs/puppet/5.5/deprecated_language.html\n   (file & line not available)
Notice: Compiled catalog for controller-0.localdomain in environment production in 0.32 seconds
Error: /Stage[main]/Tripleo::Profile::Base::Rabbitmq/Rabbitmq_policy[ha-all@/]: Could not evaluate: cannot parse line from list_policies:warning: the VM is running with native name encoding of latin1 which may cause Elixir to malfunction as it expects utf8. Please ensure your locale is set to UTF-8 (which can be verified by running "locale" in your shell)
Error: Failed to apply catalog: Cannot parse invalid user line: warning: the VM is running with native name encoding of latin1 which may cause Elixir to malfunction as it expects utf8. Please ensure your locale is set to UTF-8 (which can be verified by running "locale" in your shell)
Changes:
Events:
          Failure: 1
            Total: 1
Resources:
           Failed: 1
      Out of sync: 1
          Skipped: 5
            Total: 15
Time:
             File: 0.00
   Config retrieval: 0.40
   Rabbitmq policy: 1.13
         Last run: 1554283070
            Total: 2.39
Version:
           Config: 1554283068
           Puppet: 5.5.6
+ rc=1
+ set -e
+ '[' 1 -ne 2 -a 1 -ne 0 ']'
+ exit 1


Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:

Comment 2 Artem Hrechanychenko 2019-04-03 10:20:59 UTC
Version-Release number of selected component (if applicable):
RHOS_TRUNK-15.0-RHEL-8-20190402.n.0

How reproducible:


Steps to Reproduce:
1.Deploy OSP15 Overcloud

Actual results:
Failed "Run container-puppet tasks (bootstrap tasks) for step 2]"
"2019-04-03 09:17:03,843 ERROR: 44061 -- ['/usr/bin/podman', 'run', '--user', 'root', '--name', 'container-puppet-rabbit_init_tasks', '--env', 'PUPPET_TAGS=file,file_line,concat,augeas,cron,rabbitmq_policy,rabbitmq_user', '--env', 'NAME=rabbit_init_tasks', '--env', 'HOSTNAME=controller-0', '--env', 'NO_ARCHIVE=true', '--env', 'STEP=2', '--env', 'NET_HOST=true', '--log-driver', 'json-file', '--volume', '/etc/localtime:/etc/localtime:ro', '--volume', '/tmp/tmpk_m20olk:/etc/config.pp:ro', '--volume', '/etc/puppet/:/tmp/puppet-etc/:ro', '--volume', '/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro', '--volume', '/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro', '--volume', '/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro', '--volume', '/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro', '--volume', '/var/lib/config-data:/var/lib/config-data/:rw', '--volume', '/dev/log:/dev/log:rw', '--log-opt', 'path=/var/log/containers/stdouts/container-puppet-rabbit_init_tasks.log', '--security-opt', 'label=disable', '--volume', '/usr/share/openstack-puppet/modules/:/usr/share/openstack-puppet/modules/:ro', '--volume', '/var/lib/config-data/rabbitmq/etc/rabbitmq/:/etc/rabbitmq/:ro', '--volume', '/var/lib/rabbitmq:/var/lib/rabbitmq:z', '--entrypoint', '/var/lib/container-puppet/container-puppet.sh', '--net', 'host', '--volume', '/etc/hosts:/etc/hosts:ro', '--volume', '/var/lib/container-puppet/container-puppet.sh:/var/lib/container-puppet/container-puppet.sh:ro', '192.168.24.1:8787/rhosp15/openstack-rabbitmq:20190402.1'] run failed after + mkdir -p /etc/puppet",
        "+ '[' -n file,file_line,concat,augeas,cron,rabbitmq_policy,rabbitmq_user ']'",
        "+ TAGS='--tags file,file_line,concat,augeas,cron,rabbitmq_policy,rabbitmq_user'",
        "+ origin_of_time=/var/lib/config-data/rabbit_init_tasks.origin_of_time",
        "+ touch /var/lib/config-data/rabbit_init_tasks.origin_of_time",
        "+ /usr/bin/puppet apply --summarize --detailed-exitcodes --color=false --logdest syslog --logdest console --modulepath=/etc/puppet/modules:/usr/share/openstack-puppet/modules --tags file,file_line,concat,augeas,cron,rabbitmq_policy,rabbitmq_user /etc/config.pp",
        "Error: /Stage[main]/Tripleo::Profile::Base::Rabbitmq/Rabbitmq_policy[ha-all@/]: Could not evaluate: cannot parse line from list_policies:warning: the VM is running with native name encoding of latin1 which may cause Elixir to malfunction as it expects utf8. Please ensure your locale is set to UTF-8 (which can be verified by running \"locale\" in your shell)",
        "Error: Failed to apply catalog: Cannot parse invalid user line: warning: the VM is running with native name encoding of latin1 which may cause Elixir to malfunction as it expects utf8. Please ensure your locale is set to UTF-8 (which can be verified by running \"locale\" in your shell)",

Expected results:
Passed

Additional info:

Comment 5 Artem Hrechanychenko 2019-04-08 12:03:42 UTC
the same issue appears in undercloud & overcloud redeployment

Comment 7 Michele Baldessari 2019-04-09 07:09:45 UTC
*** Bug 1693855 has been marked as a duplicate of this bug. ***


Note You need to log in before you can comment on or make changes to this bug.