Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.

Bug 1365572

Summary: IPA server broken after upgrade
Product: Red Hat Enterprise Linux 7 Reporter: Nikhil Dehadrai <ndehadra>
Component: ipaAssignee: IPA Maintainers <ipa-maint>
Status: CLOSED ERRATA QA Contact: Kaleem <ksiddiqu>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 7.3CC: mharmsen, ndehadra, pvoborni, rcritten
Target Milestone: rcKeywords: Regression
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Fixed In Version: ipa-4.4.0-9.el7 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-11-04 06:00:44 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Bug Depends On: 1364071    
Bug Blocks: 1286635    
Description Flags
Console Output log none

Description Nikhil Dehadrai 2016-08-09 15:00:27 UTC
Description of problem:
IPA server broken after upgrade from 7.1 to 7.3

Version-Release number of selected component (if applicable):

How reproducible:

Steps to Reproduce:
1. Setup IPA server on RHEL 7.1 (In my case ipa-server-4.1.0-18.el7.x86_64)
2. Setup repo links for RHEL 7.3
3. Upgrade IPA server using command "yum -y update'ipa*' sssd"
4. After upgrade try accessing server UI.
5. After Upgrade try to restart IPA service using command "ipactl restart"
6. After upgrade check for ipaupgrade.log at /var/log path

Actual results:
1. After step4, Server UI is not accessible.
2. After step5, ipactl service fails:
   #[root@auto-hv-01-guest09 ~]# ipactl restart
    Unexpected error
    ImportError: No module named packages.urllib3.exceptions
3. After step6, noticed that ipaupgrade.log file is present but log is not populated.
   [root@auto-hv-01-guest09 log]# cat /var/log/ipaupgrade.log
   [root@auto-hv-01-guest09 log]# ls -l /var/log/ipaupgrade.log
   -rw-r--r--. 1 root root 0 Aug  9 09:09 /var/log/ipaupgrade.log
4. Also noticed following errors in /var/log/httpd/error.log file
   ipa: ERROR: cannot connect to 'https://auto-hv-01-guest09.testrelm.test/ipa/session/json': Internal Server Error
[root@auto-hv-01-guest09 ~]# tail -f /var/log/httpd/error_log 
[Tue Aug 09 09:43:17.417652 2016] [:error] [pid 20091] ipa: INFO: [jsonserver_session] admin@TESTRELM.TEST: env((u'api_version',), version=u'2.0'): SUCCESS
[Tue Aug 09 09:43:17.480375 2016] [:error] [pid 20092] ipa: INFO: [jsonserver_session] admin@TESTRELM.TEST: schema: CommandError
[Tue Aug 09 09:43:17.543136 2016] [:error] [pid 20091] ipa: INFO: [jsonserver_session] admin@TESTRELM.TEST: env((u'api_version',), version=u'2.0'): SUCCESS
[Tue Aug 09 09:43:17.601016 2016] [:error] [pid 20081] AH00000: sd_notifyf returned an error -111
[Tue Aug 09 09:43:17.696469 2016] [:error] [pid 20092] ipa: INFO: [jsonserver_session] admin@TESTRELM.TEST: schema: CommandError
[Tue Aug 09 09:43:19.183124 2016] [:error] [pid 17285] ipa: ERROR: Failed to start IPA: No module named packages.urllib3.exceptions
[Tue Aug 09 09:43:19.191890 2016] [:error] [pid 17284] ipa: ERROR: Failed to start IPA: No module named packages.urllib3.exceptions

Expected results:
Upgrade should be successful with no errors and having accessible server UI. 
Respective upgrade log should be updated correctly.
IPA service should be restarted successfully

Additional Information:
This issue was not noticed for upgrade from 7.2 to 7.3. (In my case 7.2GA to 7.3)

Comment 3 Petr Vobornik 2016-08-09 16:58:24 UTC
Seems to me as duplicate of bug 1364071. Nikhil, could you retest with pki-core-10.3.3-5.el7

Comment 4 Nikhil Dehadrai 2016-08-10 09:05:21 UTC
Hi Petr,

After updating the pki package, I am still seeing the same issue. Another thing I noticed is some errors related to "sd_notify" under httpd/error_log file.

I am attaching the console output log for reference.

Let me know if you need anymore details.

Comment 5 Nikhil Dehadrai 2016-08-10 09:05:57 UTC
Created attachment 1189515 [details]
Console Output log

Console Output log

Comment 6 Petr Vobornik 2016-08-17 17:03:44 UTC
Nikhil, could you paste output of:

rpm -qa python-urllib3 python-requests

Comment 13 Petr Vobornik 2016-08-23 12:06:53 UTC
pki-core spec needs to be raise according to comment 12

Comment 14 Petr Vobornik 2016-08-23 12:10:14 UTC
*** Bug 1365507 has been marked as a duplicate of this bug. ***

Comment 15 Petr Vobornik 2016-08-24 07:21:07 UTC
pki build with the spec change exists, so IPA should raise requires to: pki-core-10.3.3-7.el7

Moving to POST to indicate that no patch is needed.

Comment 17 Nikhil Dehadrai 2016-09-07 09:04:23 UTC
IPA Server version: ipa-server-4.4.0-9.el7.x86_64

Verified the bug bug on the basis of following points:
1. Verified that upgrade of IPA server setup on RHEL 7.1 to RHEL 7.3 is successful.
2. Verified that the error messages are observed inside "/var/log/ipaupgrade.log".
3. Verified that no error messages are observed inside "/var/log/httpd/error_log"
4. Verified that the server UI is accessible after the upgrade.
5. Verified that services can be run for "ipactl restart" and "ipactl status" successfully.
6. Also verified that "kinit admin" command runs successfully after the upgrade.

Thus on the basis of above observations marking the status of bug to "VERIFIED".

Comment 20 errata-xmlrpc 2016-11-04 06:00:44 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.