Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 1686837 - Following installation, getting Error from server (InternalError): Internal error occurred: Unexpected response: 400 [NEEDINFO]
Summary: Following installation, getting Error from server (InternalError): Internal e...
Keywords:
Status: CLOSED INSUFFICIENT_DATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Installer
Version: 3.10.0
Hardware: x86_64
OS: Linux
unspecified
high
Target Milestone: ---
: ---
Assignee: Scott Dodson
QA Contact: Johnny Liu
URL:
Whiteboard:
Depends On:
Blocks: 1542093
TreeView+ depends on / blocked
 
Reported: 2019-03-08 13:04 UTC by Dana Lane
Modified: 2019-03-14 13:33 UTC (History)
1 user (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-03-14 13:33:55 UTC
Target Upstream Version:
sdodson: needinfo? (dlane)


Attachments (Terms of Use)
Ansible log file showing error (deleted)
2019-03-08 13:04 UTC, Dana Lane
no flags Details

Description Dana Lane 2019-03-08 13:04:09 UTC
Created attachment 1542113 [details]
Ansible log file showing error

Description of problem:

Version-Release number of the following components:
rpm -q openshift-ansible = openshift-ansible-3.10.111-1.git.0.84b2b69.el7.noarch
rpm -q ansible = ansible-2.4.6.0-1.el7ae.noarch
ansible --version = ansible 2.4.6.0

How reproducible:
Same error 3 consecutive installs. Issuing a "master-restart api" fixed the problem, but this is problematic for automated installs and testing. The ansible installer is wrapped in our own ansible automation, the log starts with our automation getting the VMs prepared for the OCP install, then the end of the log is the OCP ansible installer. This completes without error, but when trying to log in following the installer, via our automation, this results in the failure. The configuration is currently in the failed state, without the restart.

Steps to Reproduce:
1. Run the ansible installer
2. Attempt to create a new user with the clu
3.

Actual results:
2019-03-07 16:20:47,580 p=32363 u=dlane |  TASK [ocp_install : Add cluster-admin role to admin user] **************************************************************************************
2019-03-07 16:20:47,580 p=32363 u=dlane |  task path: /home/bos/dlane/Projects/CI/working/scenario/css_openshift/config/ansible/roles/ocp_install/tasks/main.yml:37
2019-03-07 16:20:49,335 p=32363 u=dlane |  changed: [dhcp-19-231-183.css.lab.eng.bos.redhat.com] => {"changed": true, "cmd": ["oadm", "policy", "add-cluster-role-to-user", "cluster-admin", "admin"], "delta": "0:00:00.497931", "end": "2019-03-07 16:20:49.164888", "rc": 0, "start": "2019-03-07 16:20:48.666957", "stderr": "DEPRECATED: The 'oadm' command is deprecated, please use 'oc adm' instead.", "stderr_lines": ["DEPRECATED: The 'oadm' command is deprecated, please use 'oc adm' instead."], "stdout": "cluster role \"cluster-admin\" added: \"admin\"", "stdout_lines": ["cluster role \"cluster-admin\" added: \"admin\""]}
2019-03-07 16:20:49,386 p=32363 u=dlane |  TASK [ocp_install : Create a new OCP user] *****************************************************************************************************
2019-03-07 16:20:49,386 p=32363 u=dlane |  task path: /home/bos/dlane/Projects/CI/working/scenario/css_openshift/config/ansible/roles/ocp_install/tasks/main.yml:40
2019-03-07 16:20:51,159 p=32363 u=dlane |  fatal: [dhcp-19-231-183.css.lab.eng.bos.redhat.com]: FAILED! => {"changed": true, "cmd": ["oc", "login", "-u", "cns", "-p", "redhat"], "delta": "0:00:00.463836", "end": "2019-03-07 16:20:51.030443", "msg": "non-zero return code", "rc": 1, "start": "2019-03-07 16:20:50.566607", "stderr": "Error from server (InternalError): Internal error occurred: unexpected response: 400", "stderr_lines": ["Error from server (InternalError): Internal error occurred: unexpected response: 400"], "stdout": "", "stdout_lines": []}


Expected results:
Successful log in and new user created

Additional info:
Please attach logs from ansible-playbook with the -vvv flag
-----------
The playbook we're running;

---
# tasks file for ocp_install

- name: Install openshift-ansible
  yum:
    name: openshift-ansible
    state: installed

- name: Enable ansible logging
  lineinfile:
    dest: /etc/ansible/ansible.cfg
    insertafter: '\[defaults\]'
    line: 'log_path = /var/log/ansible.log'

- name: Copy inventory file
  template:
    src: templates/ocp_install_hosts.j2
    dest: /root/ocp_install_hosts

- name: Run OCP install playbook
  command: ansible-playbook -i {{ remote_inventory }} /usr/share/ansible/openshift-ansible/playbooks/byo/config.yml
  when: ocp_version is version_compare('3.9', '<')

- name: Run OCP prerequisites playbook
  command: ansible-playbook -i {{ remote_inventory }} /usr/share/ansible/openshift-ansible/playbooks/prerequisites.yml
  when: ocp_version is version_compare('3.9', '>=')

- name: Run OCP install playbook
  command: ansible-playbook -i {{ remote_inventory }} /usr/share/ansible/openshift-ansible/playbooks/deploy_cluster.yml
  when: ocp_version is version_compare('3.9', '>=')

- name: Fetch ansible log
  fetch:
    src: /var/log/ansible.log
    dest: ocp_install_log_{{ ansible_date_time.epoch }}

- name: Add cluster-admin role to admin user
  command: oadm policy add-cluster-role-to-user cluster-admin admin

- name: Create a new OCP user
  command: oc login -u cns -p redhat

- name: Login system:admin
  command: oc login -u system:admin

- name: Make new user a cluster admin
  command: oadm policy add-cluster-role-to-user cluster-admin cns

Comment 1 Scott Dodson 2019-03-08 13:21:09 UTC
Dana,

Why not add the restart to your role? I don't think this is anything we'd be able to address given the problem seems to arise within your custom workflow outside of the installer. If you want to pursue why it's returning a 400 I'd have to move this to the master team for them to evaluate.

--
Scott

Comment 2 Scott Dodson 2019-03-14 13:33:55 UTC
We need additional information to make any progress on this one. Closing, re-open if you can provide logs from the apiservers that indicate why they're generating a 400 error.


Note You need to log in before you can comment on or make changes to this bug.