Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.

Bug 1367037

Summary: OpenShift deployment fails at 85%
Product: Red Hat Quickstart Cloud Installer Reporter: Thom Carlin <tcarlin>
Component: Installation - OpenShiftAssignee: John Matthews <jmatthew>
Status: NEW --- QA Contact: Sudhir Mallamprabhakara <smallamp>
Severity: medium Docs Contact: Derek <dcadzow>
Priority: unspecified    
Version: 1.0CC: arubin, bthurber
Target Milestone: ---Keywords: Triaged
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description Thom Carlin 2016-08-15 11:07:08 UTC
Description of problem:

OpenShift deployment fails at 85%

Version-Release number of selected component (if applicable):

QCI-1.0-RHEL-7-20160809.t.1

How reproducible:

Uncertain

Steps to Reproduce:
1. Install/configure QCI
2. Deploy RHV and OpenShift

Actual results:

OpenShift deployment fails at 85%

Expected results:

Successful deployment

Additional info:

Task Information:

Action: Actions::Fusor::Deployment::OpenShift::InstallOSE

Exception: RuntimeError: ansible-playbook returned a non-zero exit code during installation. Please refer to the log for more information regarding the failure.

Backtrace:
/opt/theforeman/tfm/root/usr/share/gems/gems/fusor_server-1.0.0/app/lib/actions/fusor/deployment/open_shift/install_ose.rb:49:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/action.rb:506:in `block (3 levels) in execute_run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware/stack.rb:26:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware/stack.rb:26:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware.rb:17:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/action/progress.rb:30:in `with_progress_calculation'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/action/progress.rb:16:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware/stack.rb:22:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware/stack.rb:26:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware.rb:17:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.70/app/lib/actions/middleware/keep_locale.rb:11:in `block in run'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.70/app/lib/actions/middleware/keep_locale.rb:22:in `with_locale'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.70/app/lib/actions/middleware/keep_locale.rb:11:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware/stack.rb:22:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware/stack.rb:26:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware.rb:17:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware.rb:30:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware/stack.rb:22:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware/world.rb:30:in `execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/action.rb:505:in `block (2 levels) in execute_run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/action.rb:504:in `catch'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/action.rb:504:in `block in execute_run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/action.rb:419:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/action.rb:419:in `block in with_error_handling'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/action.rb:419:in `catch'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/action.rb:419:in `with_error_handling'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/action.rb:499:in `execute_run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/action.rb:260:in `execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:9:in `block (2 levels) in execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/execution_plan/steps/abstract.rb:155:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/execution_plan/steps/abstract.rb:155:in `with_meta_calculation'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:8:in `block in execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:22:in `open_action'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:7:in `execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/executors/parallel/worker.rb:15:in `block in on_message'
/opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matchers/abstract.rb:74:in `block in assigns'
/opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matchers/abstract.rb:73:in `tap'
/opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matchers/abstract.rb:73:in `assigns'
/opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:56:in `match_value'
/opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:36:in `block in match?'
/opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:35:in `each'
/opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:35:in `match?'
/opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:23:in `match'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/executors/parallel/worker.rb:12:in `on_message'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/context.rb:46:in `on_envelope'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/executes_context.rb:7:in `on_envelope'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/actor.rb:26:in `on_envelope'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/awaits.rb:15:in `on_envelope'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/sets_results.rb:14:in `on_envelope'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/buffer.rb:38:in `process_envelope'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/buffer.rb:31:in `process_envelopes?'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/buffer.rb:20:in `on_envelope'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/termination.rb:55:in `on_envelope'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/removes_child.rb:10:in `on_envelope'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/sets_results.rb:14:in `on_envelope'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/core.rb:161:in `process_envelope'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/core.rb:95:in `block in on_envelope'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/core.rb:118:in `block (2 levels) in schedule_execution'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/synchronization/mri_lockable_object.rb:38:in `block in synchronize'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/synchronization/mri_lockable_object.rb:38:in `synchronize'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/synchronization/mri_lockable_object.rb:38:in `synchronize'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/core.rb:115:in `block in schedule_execution'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/serialized_execution.rb:18:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/serialized_execution.rb:18:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/serialized_execution.rb:96:in `work'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/serialized_execution.rb:77:in `block in call_job'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:333:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:333:in `run_task'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:322:in `block (3 levels) in create_worker'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:305:in `loop'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:305:in `block (2 levels) in create_worker'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:304:in `catch'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:304:in `block in create_worker'
/opt/theforeman/tfm/root/usr/share/gems/gems/logging-1.8.2/lib/logging/diagnostic_context.rb:323:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/logging-1.8.2/lib/logging/diagnostic_context.rb:323:in `block in create_with_logging_context'

Comment 1 Thom Carlin 2016-08-15 12:00:15 UTC
ansible.log excerpts:
/usr/bin/ansible-playbook -v /opt/theforeman/tfm/root/usr/share/gems/gems/fusor_server-1.0.0/lib/modules/ose_installer/playbooks/install.yml -i /usr/share/
foreman/tmp/rhv_openshift/ansible.hosts
2016-08-14 14:39:39,409 p=34332 u=foreman |  failed: [<<openshift_master_node_fqdn>>] => {"changed": true, "cmd": "atomic-openshift-installer -u -c /tmp/atomic-openshift-installer.answers.cfg.yml install", "delta": "0:24:07.233699", "end": "2016-08-14 18:39:39.032650", "rc": 1, "start": "2016-08-14 18:15:31.798951", "warnings": []}
2016-08-14 14:39:39,409 p=34332 u=foreman |  stderr:  [WARNING]: It is unnecessary to use '{{' in conditionals, leave variables in
loop expressions bare.
[...]
<<master_fqdn>> : ok=400  changed=73   unreachable=0    failed=1   
Error from server: nodes "r<<master_fqdn>>" cannot be updated: the object has been modified; please apply your changes to the latest version and try again

FATAL: all hosts have already failed -- aborting
PLAY RECAP ******************************************************************** 
2016-08-14 14:39:39,411 p=34332 u=foreman |  <<master_fqdn>> : ok=3    changed=2    unreachable=0    failed=1   
2016-08-14 14:39:39,780 p=39940 u=foreman |   
2016-08-14 14:39:39,780 p=39940 u=foreman |  /usr/bin/ansible-playbook -v /opt/theforeman/tfm/root/usr/share/gems/gems/fusor_server-1.0.0/lib/modules/ose_installer/playbooks/install.yml -i /usr/share/
foreman/tmp/<<deployment_name>>/ansible.hosts
2016-08-14 14:39:39,780 p=39940 u=foreman |   
2016-08-14 14:39:39,807 p=39940 u=foreman |  PLAY [Prepare and execute atomic-openshift-installer] ************************* 
2016-08-14 14:39:39,807 p=39940 u=foreman |  GATHERING FACTS *************************************************************** 
2016-08-14 14:39:43,231 p=39940 u=foreman |  ok: [<<master_fqdn>>]
2016-08-14 14:39:43,231 p=39940 u=foreman |  TASK: [scp generated ansible inventory file] ********************************** 
2016-08-14 14:39:43,448 p=39940 u=foreman |  ok: [<<master_fqdn>>] => {"changed": false, "checksum": "7ace2defbc2d33de0ea147b8e92140e691a20361", "dest": "/etc/an
sible/hosts", "gid": 0, "group": "root", "mode": "0775", "owner": "root", "path": "/etc/ansible/hosts", "secontext": "system_u:object_r:etc_t:s0", "size": 2005, "state": "file", "uid": 0}
2016-08-14 14:39:43,448 p=39940 u=foreman |  TASK: [scp atomic-openshift-installer.answers.cfg.yml] ************************ 
2016-08-14 14:39:43,672 p=39940 u=foreman |  changed: [<<master_fqdn>>] => {"changed": true, "checksum": "858e96d52210cfb2f687fb6b37fa76431f031682", "dest": "/tm
p/atomic-openshift-installer.answers.cfg.yml", "gid": 0, "group": "root", "md5sum": "2d4c06ab462a3d4d2c3a4decbbae35b9", "mode": "0775", "owner": "cloudsuite-install", "secontext": "unconfined_u:object
_r:user_home_t:s0", "size": 2073, "src": "/home/cloudsuite-install/.ansible/tmp/ansible-tmp-1471199983.51-4475421010782/source", "state": "file", "uid": 1000}
2016-08-14 14:39:43,673 p=39940 u=foreman |  TASK: [execute atomic-openshift-installer] ************************************ 
2016-08-14 14:39:54,537 p=39940 u=foreman |  failed: [<<master_fqdn>>] => {"changed": true, "cmd": "atomic-openshift-installer -u -c /tmp/atomic-openshift-instal
ler.answers.cfg.yml install", "delta": "0:00:10.716404", "end": "2016-08-14 18:39:54.269910", "rc": 1, "start": "2016-08-14 18:39:43.553506", "warnings": []}

Comment 2 Thom Carlin 2016-08-15 12:06:52 UTC
/usr/bin/ansible-playbook -v /opt/theforeman/tfm/root/usr/share/gems/gems/fusor_server-1.0.0/lib/modules/ose_installer/playbooks/install.yml -i /usr/share/foreman/tmp/rhv_openshift/ansible.hosts

PLAY [Prepare and execute atomic-openshift-installer] ************************* 

GATHERING FACTS *************************************************************** 
ok: [<<master_fqdn>>]

TASK: [scp generated ansible inventory file] ********************************** 
ok: [<<master_fqdn>>m] => {"changed": false, "checksum": "7ace2defbc2d33de0ea147b8e92140e691a20361", "dest": "/etc/ansible/hosts", "gid": 0, "group": "root", "mode": "0775", "owner": "root", "path": "/etc/ansible/hosts", "secontext": "system_u:object_r:etc_t:s0", "size": 2005, "state": "file", "uid": 0}

TASK: [scp atomic-openshift-installer.answers.cfg.yml] ************************ 
ok: [<<master_fqdn>>] => {"changed": false, "checksum": "858e96d52210cfb2f687fb6b37fa76431f031682", "dest": "/tmp/atomic-openshift-installer.answers.cfg.yml", "gid": 0, "group": "root", "mode": "0775", "owner": "cloudsuite-install", "path": "/tmp/atomic-openshift-installer.answers.cfg.yml", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 2073, "state": "file", "uid": 1000}

TASK: [execute atomic-openshift-installer] ************************************ 
failed: [<<master_fqdn>>] => {"changed": true, "cmd": "atomic-openshift-installer -u -c /tmp/atomic-openshift-installer.answers.cfg.yml install", "delta": "0:00:12.030322", "end": "2016-08-15 12:02:21.501588", "rc": 1, "start": "2016-08-15 12:02:09.471266", "warnings": []}
stdout: *** Installation Summary ***

Hosts:
- <<each_node_fqdn>>
  - OpenShift Node (Dedicated)
- <<master_fqdn>>
  - OpenShift Master
  - OpenShift Node (Unscheduled)
  - Etcd (Embedded)

Total OpenShift Masters: 1
Total OpenShift Nodes: 6

NOTE: Add a total of 3 or more Masters to perform an HA installation.

Gathering information from hosts...
Installed environment detected.
<<each_node_fqdn>> is already an OpenShift Node
<<master_fqdn>> is already an OpenShift Master
Installed environment detected and no additional nodes specified: aborting. If you want a fresh install, use `atomic-openshift-installer install --force`

FATAL: all hosts have already failed -- aborting

PLAY RECAP ******************************************************************** 
           to retry, use: --limit @/root/install.retry

<<master_fqdn>> : ok=3    changed=0    unreachable=0    failed=1

Comment 3 Thom Carlin 2016-08-15 12:59:28 UTC
/tmp/ansible.log on OpenShift master [client] excerpts:
/bin/ansible-playbook --inventory-file=/tmp/hosts /usr/share/ansible/openshift-ansible//playbooks/byo/openshift_facts.yml
}
2016-08-14 18:16:37,724 p=21695 u=root |  PLAY RECAP ******************************************************************** 
2016-08-14 18:16:37,724 p=21695 u=root |  localhost                  : ok=8    changed=0    unreachable=0    failed=0   
2016-08-14 18:16:37,725 p=21695 u=root |  <<master_fqdn>> : ok=11   changed=1    unreachable=0    failed=0   
2016-08-14 18:16:37,725 p=21695 u=root |  <<each_node_fqdn>> : ok=11   changed=2    unreachable=0    failed=0   
[...]

/bin/ansible-playbook --inventory-file=/tmp/hosts /usr/share/ansible/openshift-ansible/playbooks/byo/openshift-cluster/config.yml
2016-08-14 18:16:38,098 p=22714 u=root |   
2016-08-14 18:16:38,283 p=22714 u=root |   [WARNING]: It is unnecessary to use '{{' in conditionals, leave variables in
loop expressions bare.
[...]
2016-08-14 18:39:35,506 p=22714 u=root |  TASK: [openshift_manage_node | Set node schedulability] *********************** 
2016-08-14 18:39:36,611 p=22714 u=root |  changed: [<<master_fqdn>>] = <<big_string1>>
2016-08-14 18:39:37,329 p=22714 u=root |  changed: [<<master_fqdn>>] => <<big_string2>>
2016-08-14 18:39:37,722 p=22714 u=root |  changed: [<<master_fqdn>>] => <<big_string3>>
2016-08-14 18:39:38,118 p=22714 u=root |  changed: [<<master_fqdn>>] => <<big_string4>>
2016-08-14 18:39:38,530 p=22714 u=root |  changed: [<<master_fqdn>>] => <<big_string5>>
2016-08-14 18:39:38,977 p=22714 u=root |  failed: [<<master_fqdn>>] => <<big_string6>>
 Error from server: nodes "<<master_fqdn>>" cannot be updated: the object has been modified; please apply your changes to the latest version and try again
2016-08-14 18:39:38,980 p=22714 u=root |  FATAL: all hosts have already failed -- aborting
2016-08-14 18:39:38,980 p=22714 u=root |  PLAY RECAP ******************************************************************** 
2016-08-14 18:39:38,981 p=22714 u=root |             to retry, use: --limit @/root/config.retry

2016-08-14 18:39:38,981 p=22714 u=root |  localhost                  : ok=17   changed=0    unreachable=0    failed=0   
2016-08-14 18:39:38,981 p=22714 u=root |  <<master_fqdn>> : ok=400  changed=73   unreachable=0    failed=1   
2016-08-14 18:39:38,981 p=22714 u=root |  <<each_node_fqdn>> : ok=152  changed=34   unreachable=0    failed=0