Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 1518376 - ipa-server-install fails to install
Summary: ipa-server-install fails to install
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: ipa
Version: 7.4
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: rc
: ---
Assignee: IPA Maintainers
QA Contact: ipa-qe
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-11-28 18:45 UTC by Steve Dickson
Modified: 2018-01-30 18:50 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-01-30 18:50:56 UTC


Attachments (Terms of Use)

Description Steve Dickson 2017-11-28 18:45:23 UTC
Description of problem:
On an updated rhel7.4-Z VM I tried to install an ipa-server that
failed with the following errors:

Configuring certificate server (pki-tomcatd). Estimated time: 3 minutes
  [1/29]: configuring certificate server instance
  [2/29]: exporting Dogtag certificate store pin
  [3/29]: stopping certificate server instance to update CS.cfg
  [4/29]: backing up CS.cfg
  [5/29]: disabling nonces
  [6/29]: set up CRL publishing
  [7/29]: enable PKIX certificate path discovery and validation
  [8/29]: starting certificate server instance
  [9/29]: configure certmonger for renewals
  [10/29]: requesting RA certificate from CA
  [error] RuntimeError: request timed out
ipa.ipapython.install.cli.install_tool(CompatServerMasterInstall): ERROR    request timed out
ipa.ipapython.install.cli.install_tool(CompatServerMasterInstall): ERROR    The ipa-server-install command failed. See /var/log/ipaserver-install.log for more i

Version-Release number of selected component (if applicable):
ipa-server-4.5.0-21.el7_4.2.2.x86_64

How reproducible:


Steps to Reproduce:
1.Type ipa-server-install 


Additional info:

ipatserver-install.log
    http://people.redhat.com/steved/.tmp/ipaserver-install.log

ipaserver-screenshot
    http://people.redhat.com/steved/.tmp/ipaserver-screenshot.txt

Comment 2 Steve Dickson 2017-11-28 18:59:10 UTC
It appears this was the problem... 

WARNING: Your system is running out of entropy, you may experience long delays

The rngd service failed:

Starting Hardware RNG Entropy Gatherer Daemon...
read error
hwrng: no available rng
Unable to open file: /dev/tpm0
can't open any entropy source
Maybe RNG device modules are not loaded
rngd.service: main process exited, code=exited, status=1/FAILURE
Unit rngd.service entered failed state.
rngd.service failed.

Comment 3 Steve Dickson 2017-11-28 19:08:13 UTC

*** This bug has been marked as a duplicate of bug 1464200 ***

Comment 4 Steve Dickson 2017-12-05 19:08:57 UTC
I'm reopening this up because I've had the same problem in RHEL7.4 and
now RHEL 7.5 nightly and with these last two times there was
no ngd.service failure... 

Plus the system is been eaten alive by these processes now 
and when I reboot (if I don't uninstall the ipaserver).

14764 ?        R      0:33 /usr/bin/python2 -E /usr/libexec/certmonger/dogtag-ipa-ca-renew-agent-submit
14765 ?        R      0:34 /usr/bin/python2 -E /usr/libexec/certmonger/dogtag-ipa-ca-renew-agent-submit
14766 ?        R      0:33 /usr/bin/python2 -E /usr/libexec/certmonger/dogtag-ipa-ca-renew-agent-submit
14767 ?        R      0:33 /usr/bin/python2 -E /usr/libexec/certmonger/dogtag-ipa-ca-renew-agent-submit
14768 ?        R      0:33 /usr/bin/python2 -E /usr/libexec/certmonger/dogtag-ipa-ca-renew-agent-submit
14769 ?        R      0:33 /usr/bin/python2 -E /usr/libexec/certmonger/dogtag-ipa-ca-renew-agent-submit
14770 ?        R      0:32 /usr/bin/python2 -E /usr/libexec/certmonger/dogtag-ipa-ca-renew-agent-submit
14771 ?        R      0:32 /usr/bin/python2 -E /usr/libexec/certmonger/dogtag-ipa-ca-renew-agent-submit

Here is the screen shot:
Configuring certificate server (pki-tomcatd). Estimated time: 3 minutes
  [1/29]: configuring certificate server instance
  [2/29]: exporting Dogtag certificate store pin
  [3/29]: stopping certificate server instance to update CS.cfg
  [4/29]: backing up CS.cfg
  [5/29]: disabling nonces
  [6/29]: set up CRL publishing
  [7/29]: enable PKIX certificate path discovery and validation
  [8/29]: starting certificate server instance
  [9/29]: configure certmonger for renewals
  [10/29]: requesting RA certificate from CA
  [error] RuntimeError: request timed out
ipa.ipapython.install.cli.install_tool(CompatServerMasterInstall): ERROR    request timed out
ipa.ipapython.install.cli.install_tool(CompatServerMasterInstall): ERROR    The ipa-server-install command failed. See /var/log/ipaserver-install.log for more information

The updated log are under
    http://people.redhat.com/steved/.bz1518376/

Comment 5 Florence Blanc-Renaud 2017-12-06 14:22:36 UTC
Hi,

can you check on your system if there is a file /var/run/ipa/renewal.lock and what is its content?

On a clean system, this file is either not present or contains the following:
# cat /var/run/ipa/renewal.lock
[lock]
locked = 0

The file may be a left-over from the failing installation, and if it contains locked=1, then the request for a RA certificate will wait for the lock to be released and exit on timeout.

The workaround in this case is either to wait for long enough, or to delete the file and re-launch ipa-server-install.

Note that you may need to remove the cert requests manually after a failing installation:
ipa-server-install --uninstall -U
getcert list
^^ If this command returns requests, you must run getcert stop-tracking <ID> for each remaining request.

Comment 6 Steve Dickson 2017-12-06 16:16:13 UTC
(In reply to Florence Blanc-Renaud from comment #5)
> Hi,
> 
> can you check on your system if there is a file /var/run/ipa/renewal.lock
> and what is its content?
> 
> On a clean system, this file is either not present or contains the following:
> # cat /var/run/ipa/renewal.lock
> [lock]
> locked = 0
The file exists... 
ipa# cat /var/run/ipa/renewal.lock
[lock]
locked = 0

> 
> The file may be a left-over from the failing installation, and if it
> contains locked=1, then the request for a RA certificate will wait for the
> lock to be released and exit on timeout.
> 
> The workaround in this case is either to wait for long enough, or to delete
> the file and re-launch ipa-server-install.
I'll try removing the file and reinstalling.

> 
> Note that you may need to remove the cert requests manually after a failing
> installation:
> ipa-server-install --uninstall -U
> getcert list
> ^^ If this command returns requests, you must run getcert stop-tracking <ID>
> for each remaining request.
Its appears this is good....

ipa# getcert list
Number of certificates and requests being tracked: 0.

Thanks!

Comment 7 Steve Dickson 2017-12-06 18:25:37 UTC
I tried again... with the same timeout

(In reply to Steve Dickson from comment #6)
> (In reply to Florence Blanc-Renaud from comment #5)
> > Hi,
> > 
> > can you check on your system if there is a file /var/run/ipa/renewal.lock
> > and what is its content?
> > 
> > On a clean system, this file is either not present or contains the following:
> > # cat /var/run/ipa/renewal.lock
> > [lock]
> > locked = 0
> The file exists... 
> ipa# cat /var/run/ipa/renewal.lock
> [lock]
> locked = 0
This file exists in the above state.

> 
> > 
> > Note that you may need to remove the cert requests manually after a failing
> > installation:
> > ipa-server-install --uninstall -U
> > getcert list
> > ^^ If this command returns requests, you must run getcert stop-tracking <ID>
> > for each remaining request.
> Its appears this is good....
> 
> ipa# getcert list
> Number of certificates and requests being tracked: 0.
This now has something in it... 
ipa# getcert list
Number of certificates and requests being tracked: 1.
Request ID '20171206164006':
	status: MONITORING
	stuck: no
	key pair storage: type=FILE,location='/var/lib/ipa/ra-agent.key'
	certificate: type=FILE,location='/var/lib/ipa/ra-agent.pem'
	CA: dogtag-ipa-ca-renew-agent
	issuer: CN=Certificate Authority,O=BOSTON.DEVEL.REDHAT.COM
	subject: CN=IPA RA,O=BOSTON.DEVEL.REDHAT.COM
	expires: 2019-11-26 16:40:30 UTC
	key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
	eku: id-kp-serverAuth,id-kp-clientAuth
	pre-save command: /usr/libexec/ipa/certmonger/renew_ra_cert_pre
	post-save command: /usr/libexec/ipa/certmonger/renew_ra_cert
	track: yes
	auto-renew: yes

Are you able to install a new RHEL7.5 ipa-server?

Comment 8 Florence Blanc-Renaud 2017-12-07 13:40:42 UTC
Hi,

from the logs provided, it looks like your system is slow delivering the certificate:
2017-12-05T18:45:51Z DEBUG certmonger request is in state dbus.String(u'GENERATING_KEY_PAIR', variant_level=1)
...
2017-12-05T18:51:29Z DEBUG certmonger request is in state dbus.String(u'POST_SAVED_CERT', variant_level=1)
2017-12-05T18:51:37Z DEBUG Traceback (most recent call last):

This means that it needs more that 5min to deliver the cert, and is above the default timeout (300s).

You can configure the timeout by creating a file /etc/ipa/installer.conf with the following content:

# cat /etc/ipa/installer.conf
[global]
startup_timeout=800

(the timeout is in seconds and can be tuned for your system)
and retry ipa-server-install with this setting.

Please let me know if it solves your problem.

Comment 9 Steve Dickson 2017-12-07 15:55:31 UTC
Here is some strangeness that might interesting... 

In perpetration doing the install with the bumped up
timeout value I did a uninstall and got the following 
error:

ipa# ipa-server-install --uninstall -U
Shutting down all IPA services
Unconfiguring ntpd
Configuring certmonger to stop tracking system certificates for KRA
Configuring certmonger to stop tracking system certificates for CA
Unconfiguring CA
Unconfiguring krb5kdc
Unconfiguring kadmin
Unconfiguring directory server
ipa         : ERROR    Unable to find server cert nickname in /etc/dirsrv/slapd-BOSTON-DEVEL-REDHAT-COM/dse.ldif
Removing IPA client configuration
Removing Kerberos service principals from /etc/krb5.keytab
Disabling client Kerberos and LDAP configurations
Redundant SSSD configuration file /etc/sssd/sssd.conf was moved to /etc/sssd/sssd.conf.deleted
nscd daemon is not installed, skip configuration
nslcd daemon is not installed, skip configuration
Client uninstall complete.
The ipa-client-install command was successful

Then when I rebooted I got the following error:

[FAILED] Failed to start 389 Directory Server BOSTON-DEVEL-REDHAT-COM..
See 'systemctl status dirsrv@BOSTON-DEVEL-REDHAT-COM.service' for details.

So maybe something is not being cleaned up correctly??

Comment 10 Steve Dickson 2017-12-07 16:05:34 UTC
(In reply to Steve Dickson from comment #9)
>
> Then when I rebooted I got the following error:
> 
> [FAILED] Failed to start 389 Directory Server BOSTON-DEVEL-REDHAT-COM..
> See 'systemctl status dirsrv@BOSTON-DEVEL-REDHAT-COM.service' for details.
> 
> So maybe something is not being cleaned up correctly??

Note, even when I yum remove ipa-server, I still got this error.

Comment 11 Florence Blanc-Renaud 2017-12-07 17:16:46 UTC
You may try
systemctl disable dirsrv@BOSTON-DEVEL-REDHAT-COM.service

(the service unit was probably not removed by the uninstall).
When I see errors in the uninstall, I usually relaunch the same command a second time to make sure that everything was removed.

Comment 12 Steve Dickson 2017-12-07 18:00:44 UTC
Well it did get a little further by upping the timeout

onfiguring certificate server (pki-tomcatd). Estimated time: 3 minutes
  [1/29]: configuring certificate server instance
  [2/29]: exporting Dogtag certificate store pin
  [3/29]: stopping certificate server instance to update CS.cfg
  [4/29]: backing up CS.cfg
  [5/29]: disabling nonces
  [6/29]: set up CRL publishing
  [7/29]: enable PKIX certificate path discovery and validation
  [8/29]: starting certificate server instance
  [9/29]: configure certmonger for renewals
  [10/29]: requesting RA certificate from CA
  [11/29]: setting up signing cert profile
  [12/29]: setting audit signing renewal to 2 years
  [13/29]: restarting certificate server
  [error] CalledProcessError: Command '/bin/systemctl restart pki-tomcatd@pki-tomcat.service' returned non-zero exit status 1
ipa.ipapython.install.cli.install_tool(CompatServerMasterInstall): ERROR    Command '/bin/systemctl restart pki-tomcatd@pki-tomcat.service' returned non-zero exit status 1
ipa.ipapython.install.cli.install_tool(CompatServerMasterInstall): ERROR    The ipa-server-install command failed. See /var/log/ipaserver-install.log for more information

The install log
   http://people.redhat.com/steved/.bz1518376/ipaserver-install.log

Comment 13 Steve Dickson 2017-12-07 18:04:46 UTC
(In reply to Florence Blanc-Renaud from comment #11)
> You may try
> systemctl disable dirsrv@BOSTON-DEVEL-REDHAT-COM.service
> 
> (the service unit was probably not removed by the uninstall).
> When I see errors in the uninstall, I usually relaunch the same command a
> second time to make sure that everything was removed.

When I removed the 389-ds-base package, the ipa-server
also got removed. So when I installed the ipa-server 
the 389-ds-base was install. 

I'll go ahead and uninstall the server (twice if 
necessary), disable  dirsrv@BOSTON-DEVEL-REDHAT-COM,
reboot and then try another install with the
bumped up timeout

Thanks for the help!

Comment 14 Steve Dickson 2017-12-07 18:33:48 UTC
(In reply to Florence Blanc-Renaud from comment #11)
> 
> (the service unit was probably not removed by the uninstall).
> When I see errors in the uninstall, I usually relaunch the same command a
> second time to make sure that everything was removed.
Here is what it looks when I do the uninstall twice
(I'm thinking this is all expected)

ipa# ipa-server-install --uninstall -U
WARNING:
IPA server is not configured on this system. If you want to install the
IPA server, please install it using 'ipa-server-install'.

WARNING: Failed to connect to Directory Server to find information about
replication agreements. Uninstallation will continue despite the possible
existing replication agreements.

If this server is the last instance of CA,
KRA, or DNSSEC master, uninstallation may result in data loss.
Shutting down all IPA services
Configuring certmonger to stop tracking system certificates for KRA
Configuring certmonger to stop tracking system certificates for CA
Removing IPA client configuration
IPA client is not configured on this system.
The ipa-client-install command failed. See /var/log/ipaclient-uninstall.log for more information

http://people.redhat.com/steved/.bz1518376/ipaclient-uninstall.log

Comment 15 Steve Dickson 2017-12-08 18:42:41 UTC
(In reply to Steve Dickson from comment #13)
> (In reply to Florence Blanc-Renaud from comment #11)
> > You may try
> > systemctl disable dirsrv@BOSTON-DEVEL-REDHAT-COM.service
> > 
> > (the service unit was probably not removed by the uninstall).
> > When I see errors in the uninstall, I usually relaunch the same command a
> > second time to make sure that everything was removed.
> 
> When I removed the 389-ds-base package, the ipa-server
> also got removed. So when I installed the ipa-server 
> the 389-ds-base was install. 
> 
> I'll go ahead and uninstall the server (twice if 
> necessary), disable  dirsrv@BOSTON-DEVEL-REDHAT-COM,
> reboot and then try another install with the
> bumped up timeout
From an IRC conversion it was suggested I beef up my VM,
which I did... 8 cpus 13k memory (most I can do) but
unfortunately I got the same error as in Comment 7

Again, I'll ask has anybody had any success installing
an ipa server with the latest RHEL 7.5 bits? 

I have been trying for the last week with no success...

Comment 16 Florence Blanc-Renaud 2017-12-11 09:01:00 UTC
Hi,

I just tried to install with ipa-server 4.5.4-6.el7 on RHEL 7.5 and succeeded on our lab.
The default lab machines for RHEL 7.5 have 4 CPUs and 3GB RAM (the official doc recommends at least 2 GB of RAM and 1 GB swap space).

Comment 17 Florence Blanc-Renaud 2018-01-04 13:29:01 UTC
Hi,

were you able to retry with enough memory? Please share your logs if the installation is still failing (if the issue is still about pki-tomcatd failing to restart, we will also need the logs from /var/log/pki).

Comment 18 Steve Dickson 2018-01-12 14:10:13 UTC
(In reply to Florence Blanc-Renaud from comment #17)
> Hi,
> 
> were you able to retry with enough memory? Please share your logs if the
> installation is still failing (if the issue is still about pki-tomcatd
> failing to restart, we will also need the logs from /var/log/pki).

After I do the install the machine is eaten alive by the
dogtag-ipa-ca-renew-agent-submit process 

ipa$ ps ax | grep dogtag
 1389 ?        R      0:54 /usr/bin/python2 -E /usr/libexec/certmonger/dogtag-ipa-ca-renew-agent-submit
 1390 ?        R      0:54 /usr/bin/python2 -E /usr/libexec/certmonger/dogtag-ipa-ca-renew-agent-submit
 1391 ?        R      0:53 /usr/bin/python2 -E /usr/libexec/certmonger/dogtag-ipa-ca-renew-agent-submit
 1392 ?        R      0:53 /usr/bin/python2 -E /usr/libexec/certmonger/dogtag-ipa-ca-renew-agent-submit
 1395 ?        R      0:53 /usr/bin/python2 -E /usr/libexec/certmonger/dogtag-ipa-ca-renew-agent-submit
 1399 ?        R      0:53 /usr/bin/python2 -E /usr/libexec/certmonger/dogtag-ipa-ca-renew-agent-submit
 1400 ?        R      0:54 /usr/bin/python2 -E /usr/libexec/certmonger/dogtag-ipa-ca-renew-agent-submit
 1401 ?        R      0:53 /usr/bin/python2 -E /usr/libexec/certmonger/dogtag-ipa-ca-renew-agent-submit
 1402 ?        R      0:53 /usr/bin/python2 -E /usr/libexec/certmonger/dogtag-ipa-ca-renew-agent-submit --reuse-existing
 1403 ?        R      0:53 /usr/bin/python2 -E /usr/libexec/certmonger/dogtag-ipa-ca-renew-agent-submit --reuse-existing
 1405 ?        R      0:54 /usr/bin/python2 -E /usr/libexec/certmonger/dogtag-ipa-ca-renew-agent-submit --reuse-existing
 1406 ?        R      0:53 /usr/bin/python2 -E /usr/libexec/certmonger/dogtag-ipa-ca-renew-agent-submit --reuse-existing
 1407 ?        R      0:53 /usr/bin/python2 -E /usr/libexec/certmonger/dogtag-ipa-ca-renew-agent-submit --reuse-existing
 1409 ?        R      0:53 /usr/bin/python2 -E /usr/libexec/certmonger/dogtag-ipa-ca-renew-agent-submit --reuse-existing
 1410 ?        R      0:53 /usr/bin/python2 -E /usr/libexec/certmonger/dogtag-ipa-ca-renew-agent-submit --reuse-existing
 1411 ?        R      0:53 /usr/bin/python2 -E /usr/libexec/certmonger/dogtag-ipa-ca-renew-agent-submit --reuse-existing

ipa$ uptime
 09:04:31 up 7 min,  1 user,  load average: 19.57, 15.56, 7.53

Comment 21 Rob Crittenden 2018-01-29 16:06:01 UTC
I'm not sure how to proceed with this. It isn't something we've been able to reproduce independently. Given this is a VM is there a chance you can try provisioning a fresh one and trying there?

Comment 22 Steve Dickson 2018-01-30 18:50:56 UTC
(In reply to Rob Crittenden from comment #21)
> I'm not sure how to proceed with this. It isn't something we've been able to
> reproduce independently. Given this is a VM is there a chance you can try
> provisioning a fresh one and trying there?

I was able to do an install on a different machine and
different vm... I think we can close this bz


Note You need to log in before you can comment on or make changes to this bug.