Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 1358752 - ipa-ca-install fails on replica when IPA server is converted from CA-less to CA-full
Summary: ipa-ca-install fails on replica when IPA server is converted from CA-less to ...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: ipa
Version: 7.3
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: rc
: ---
Assignee: IPA Maintainers
QA Contact: Michal Reznik
URL:
Whiteboard:
Depends On: 1365858
Blocks: 1415158
TreeView+ depends on / blocked
 
Reported: 2016-07-21 12:16 UTC by Abhijeet Kasurde
Modified: 2017-08-01 09:39 UTC (History)
16 users (show)

Fixed In Version: ipa-4.5.0-1.el7
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1415158 (view as bug list)
Environment:
Last Closed: 2017-08-01 09:39:54 UTC


Attachments (Terms of Use)
ipareplica1-ca_debug.log (deleted)
2016-07-21 12:24 UTC, Abhijeet Kasurde
no flags Details
ipamaster1-tomcat.log (deleted)
2016-07-21 12:25 UTC, Abhijeet Kasurde
no flags Details
ipareplica1-tomcat.log (deleted)
2016-07-21 12:26 UTC, Abhijeet Kasurde
no flags Details
ipamaster1-ca_debug.log (deleted)
2016-07-21 12:27 UTC, Abhijeet Kasurde
no flags Details
ipamaster1-ipaclient-install.log (deleted)
2016-07-21 12:29 UTC, Abhijeet Kasurde
no flags Details
ipamaster1-ipareplica-conncheck.log (deleted)
2016-07-21 12:30 UTC, Abhijeet Kasurde
no flags Details
ipamaster1-ipareplica-install.log (deleted)
2016-07-21 12:30 UTC, Abhijeet Kasurde
no flags Details
ipamaster1-ipaserver-ca-install.log (deleted)
2016-07-21 12:31 UTC, Abhijeet Kasurde
no flags Details
ipareplica1-ipaclient-install.log (deleted)
2016-07-21 12:31 UTC, Abhijeet Kasurde
no flags Details
ipareplica1-ipaserver-ca-install.log (deleted)
2016-07-21 12:37 UTC, Abhijeet Kasurde
no flags Details
ipareplica1-ipaserver-install.log (deleted)
2016-07-21 12:37 UTC, Abhijeet Kasurde
no flags Details
pkispawn_ca.cfg (deleted)
2016-07-21 12:39 UTC, Abhijeet Kasurde
no flags Details
ipareplica1-ca_debug.log (deleted)
2016-07-22 03:44 UTC, Abhijeet Kasurde
no flags Details
ipamaster1-ipaserver-ca-install.log (deleted)
2016-07-22 03:48 UTC, Abhijeet Kasurde
no flags Details
ipareplica1-ipaserver-ca-install.log (deleted)
2016-07-22 03:50 UTC, Abhijeet Kasurde
no flags Details
pki-ca-spawn (deleted)
2017-05-16 11:29 UTC, Michal Reznik
no flags Details
ca_debug20170516 (deleted)
2017-05-16 11:30 UTC, Michal Reznik
no flags Details
journalctl_pki-tomcat20170516 (deleted)
2017-05-16 11:31 UTC, Michal Reznik
no flags Details
20170517_master_journaclt_pki-tomcat (deleted)
2017-05-17 11:41 UTC, Michal Reznik
no flags Details
20170517_master_ca-debug (deleted)
2017-05-17 11:42 UTC, Michal Reznik
no flags Details
20170517_master_pki-spawn071437 (deleted)
2017-05-17 11:43 UTC, Michal Reznik
no flags Details
20170517_replica_journaclt_pki-tomcat (deleted)
2017-05-17 11:44 UTC, Michal Reznik
no flags Details
20170517_replica_pki-spawn071846 (deleted)
2017-05-17 11:45 UTC, Michal Reznik
no flags Details
20170517_replica_ca-debug (deleted)
2017-05-17 11:47 UTC, Michal Reznik
no flags Details


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2017:2304 normal SHIPPED_LIVE ipa bug fix and enhancement update 2017-08-01 12:41:35 UTC
Red Hat Bugzilla 1365858 None None None Never

Internal Links: 1365858

Description Abhijeet Kasurde 2016-07-21 12:16:55 UTC
Description of problem:
After promoting IPA server from CA-less to CA-full, ipa-ca-install fails to install CA on replica server. 


Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1. Install IPA server 

[root@ipamaster1 ca1]# ipa-server-install --ip-address $(ip addr|grep "global"|cut -d " " -f6|cut -d "/" -f1|head -n 1) -r testrelm.test -p 'Secret123' -a 'Secret123' --setup-dns --forwarder 10.65.201.89 -U --dirsrv-cert-file=./server.p12 --http-cert-file=./server.p12 --dirsrv-pin Secret123 --http-pin Secret123


2. Install CA-less replica 

[root@ipareplica1 ca1]# ipa-replica-install -U --dirsrv-cert-file=./replica.p12 --http-cert-file=./replica.p12 --dirsrv-pin Secret123 --http-pin Secret123 -P admin -w Secret123 


3. Promote IPA server from CA-less to CA-full

[root@ipamaster1 ca1]# ipa-ca-install

4. Try to promote IPA replica from CA-less to CA-full

[root@ipareplica1 ca1]# ipa-ca-install 
Directory Manager (existing master) password: 

Run connection check to master
Connection check OK
/usr/lib/python2.7/site-packages/urllib3/connection.py:251: SecurityWarning: Certificate has no `subjectAltName`, falling back to check for a `commonName` for now. This feature is being removed by major browsers and deprecated by RFC 2818. (See https://github.com/shazow/urllib3/issues/497 for details.)
  SecurityWarning
Configuring certificate server (pki-tomcatd). Estimated time: 3 minutes 30 seconds
  [1/25]: creating certificate server user
  [2/25]: creating certificate server db
  [3/25]: setting up initial replication
Starting replication, please wait until this has completed.
Update in progress, 4 seconds elapsed
Update succeeded

  [4/25]: creating installation admin user
  [5/25]: setting up certificate server
ipa.ipaserver.install.cainstance.CAInstance: CRITICAL Failed to configure CA instance: Command '/usr/sbin/pkispawn -s CA -f /tmp/tmpzHP3HH' returned non-zero exit status 1
ipa.ipaserver.install.cainstance.CAInstance: CRITICAL See the installation logs and the following files/directories for more information:
ipa.ipaserver.install.cainstance.CAInstance: CRITICAL   /var/log/pki/pki-tomcat
  [error] RuntimeError: CA configuration failed.

Your system may be partly configured.
Run /usr/sbin/ipa-server-install --uninstall to clean up.

CA configuration failed.


Actual results:
IPA replica promotion from CA-less to CA-full fails with stack trace 

Expected results:
IPA replica should be converted from CA-less to CA-full.

Additional info:

1. Excerpt from /var/log/pki/pki-tomcat/ca/debug on replica

[21/Jul/2016:07:36:45][http-bio-8443-exec-3]: Getting install token
[21/Jul/2016:07:36:47][http-bio-8443-exec-3]: Getting domain XML
[21/Jul/2016:07:36:47][http-bio-8443-exec-3]: ConfigurationUtils: getting domain info
[21/Jul/2016:07:36:47][http-bio-8443-exec-3]: ConfigurationUtils: GET https://ipamaster1.testrelm.test:443/ca/admin/ca/getDomainXML
javax.ws.rs.ProcessingException: Unable to invoke request
<snip>
</snip>
        at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
        at org.jboss.resteasy.client.jaxrs.engines.ApacheHttpClient4Engine.invoke(ApacheHttpClient4Engine.java:283)
        ... 73 more
[21/Jul/2016:07:36:47][http-bio-8443-exec-3]: Failed to obtain security domain decriptor from security domain master: javax.ws.rs.ProcessingException: Unable to invoke request

Comment 1 Abhijeet Kasurde 2016-07-21 12:24:29 UTC
Created attachment 1182459 [details]
ipareplica1-ca_debug.log

Comment 2 Abhijeet Kasurde 2016-07-21 12:25:14 UTC
Created attachment 1182461 [details]
ipamaster1-tomcat.log

Comment 3 Abhijeet Kasurde 2016-07-21 12:26:21 UTC
Created attachment 1182463 [details]
ipareplica1-tomcat.log

Comment 4 Abhijeet Kasurde 2016-07-21 12:27:33 UTC
Created attachment 1182464 [details]
ipamaster1-ca_debug.log

Comment 5 Abhijeet Kasurde 2016-07-21 12:29:49 UTC
Created attachment 1182465 [details]
ipamaster1-ipaclient-install.log

Comment 6 Abhijeet Kasurde 2016-07-21 12:30:10 UTC
Created attachment 1182466 [details]
ipamaster1-ipareplica-conncheck.log

Comment 7 Abhijeet Kasurde 2016-07-21 12:30:55 UTC
Created attachment 1182468 [details]
ipamaster1-ipareplica-install.log

Comment 8 Abhijeet Kasurde 2016-07-21 12:31:13 UTC
Created attachment 1182469 [details]
ipamaster1-ipaserver-ca-install.log

Comment 9 Abhijeet Kasurde 2016-07-21 12:31:46 UTC
Created attachment 1182470 [details]
ipareplica1-ipaclient-install.log

Comment 10 Abhijeet Kasurde 2016-07-21 12:37:06 UTC
Created attachment 1182474 [details]
ipareplica1-ipaserver-ca-install.log

Comment 11 Abhijeet Kasurde 2016-07-21 12:37:57 UTC
Created attachment 1182475 [details]
ipareplica1-ipaserver-install.log

Comment 12 Abhijeet Kasurde 2016-07-21 12:39:12 UTC
Created attachment 1182489 [details]
pkispawn_ca.cfg

Comment 13 Petr Vobornik 2016-07-21 13:08:03 UTC
seems that ipareplica1-ca_debug.log contains content of ipareplica1-tomcat.log. And ipamaster1-ipaserver-ca-install.log and ipareplica1-ipaserver-ca-install.log seems to be switched.

Could you attach proper ipareplica1-ca_debug.log? Or ideally replica's pki spawn log.

Comment 15 Abhijeet Kasurde 2016-07-22 03:44:15 UTC
Created attachment 1182691 [details]
ipareplica1-ca_debug.log

Comment 16 Abhijeet Kasurde 2016-07-22 03:48:53 UTC
Created attachment 1182692 [details]
ipamaster1-ipaserver-ca-install.log

Comment 17 Abhijeet Kasurde 2016-07-22 03:50:57 UTC
Created attachment 1182694 [details]
ipareplica1-ipaserver-ca-install.log

Comment 18 Martin Bašti 2016-07-27 12:17:34 UTC

*** This bug has been marked as a duplicate of bug 1301546 ***

Comment 19 Endi Sukma Dewata 2016-08-03 23:33:51 UTC
Hi,

I'm fixing PKI to work with the PKCS #12 file generated by Custodia. However, it looks like the PKCS #12 file does not contain the 3rd-party SSL certificate (server.p12) which needs to be trusted by PKI replica in order to communicate with PKI master via IPA's HTTP proxy. Could someone confirm which SSL certificate is used by the HTTP proxy in this particular configuration?

Also, since PKI master is configured to use an SSL connection to the DS, the PKI replica must also be configured with pki_ds_secure_connection=True. The attached ipareplica1-ipaserver-install.log indicates that that is not the case.

Comment 21 Endi Sukma Dewata 2016-08-04 00:16:44 UTC
With the new PKI packages the /var/log/pki/pki-tomcat/ca/debug shows that the master HTTP proxy is using the 3rd-party SSL certificate, not the IPA certificate generated by PKI:

[04/Aug/2016:00:21:25][http-bio-8443-exec-3]: ConfigurationUtils: GET https://master.example.com:443/ca/admin/ca/getDomainXML
...
[04/Aug/2016:00:21:25][http-bio-8443-exec-3]: Server certificate:
[04/Aug/2016:00:21:25][http-bio-8443-exec-3]:  - subject: CN=master.example.com
[04/Aug/2016:00:21:25][http-bio-8443-exec-3]:  - issuer: CN="Certificate Authority,O=EXTERNAL"
[04/Aug/2016:00:21:25][http-bio-8443-exec-3]: ERROR: UNTRUSTED_ISSUER
javax.ws.rs.ProcessingException: Unable to invoke request
        at org.jboss.resteasy.client.jaxrs.engines.ApacheHttpClient4Engine.invoke(ApacheHttpClient4Engine.java:287)
...

Also the /var/log/ipareplica-ca-install.log shows that neither the 3rd-party SSL certificate or its CA certificate is included in the PKCS #12 file:

Importing certificates from /tmp/ca.p12:
---------------
4 entries found
---------------
  Certificate ID: 7fa6641e55d9693b49207f64f5f8932a2eb3d9bb
  Serial Number: 0x1
  Nickname: caSigningCert cert-pki-ca
  Subject DN: CN=Certificate Authority,O=EXAMPLE
  Issuer DN: CN=Certificate Authority,O=EXAMPLE
  Trust Flags: u,u,u
  Has Key: true

  Certificate ID: 124db33431d3e90dfb05a17a5b8a1b65160a983c
  Serial Number: 0x5
  Nickname: auditSigningCert cert-pki-ca
  Subject DN: CN=CA Audit,O=EXAMPLE
  Issuer DN: CN=Certificate Authority,O=EXAMPLE
  Trust Flags: u,u,u
  Has Key: true

  Certificate ID: ba9330c3bedbbc0014203cd74082de1c539fa0e7
  Serial Number: 0x2
  Nickname: ocspSigningCert cert-pki-ca
  Subject DN: CN=OCSP Subsystem,O=EXAMPLE
  Issuer DN: CN=Certificate Authority,O=EXAMPLE
  Trust Flags: u,u,u
  Has Key: true

  Certificate ID: e2319cb39ff2a48e4a949685d161a712430f8d9c
  Serial Number: 0x4
  Nickname: subsystemCert cert-pki-ca
  Subject DN: CN=CA Subsystem,O=EXAMPLE
  Issuer DN: CN=Certificate Authority,O=EXAMPLE
  Trust Flags: u,u,u
  Has Key: true
---------------
Import complete
---------------
Imported certificates in /etc/pki/pki-tomcat/alias:

Certificate Nickname                                         Trust Attributes
                                                             SSL,S/MIME,JAR/XPI

caSigningCert cert-pki-ca                                    CTu,Cu,Cu
auditSigningCert cert-pki-ca                                 u,u,Pu
ocspSigningCert cert-pki-ca                                  u,u,u
subsystemCert cert-pki-ca                                    u,u,u

Comment 24 Petr Vobornik 2016-08-05 07:10:17 UTC
First ipa-ca-install uses LDAPS because of https://fedorahosted.org/freeipa/ticket/5570

The logic in IPA code is:
"""
        if not self.create_ra_agent_db and not self.clone:
            self._use_ldaps_during_spawn(config)
"""

"""
    def _use_ldaps_during_spawn(self, config, ds_cacert=paths.IPA_CA_CRT):
        config.set(self.subsystem, "pki_ds_ldaps_port", "636")
        config.set(self.subsystem, "pki_ds_secure_connection", "True")
        config.set(self.subsystem, "pki_ds_secure_connection_ca_pem_file",
                   ds_cacert)
"""

Does it mean that in CA-less -> CA scenario every CA replica will use LDAPS instead of TLS over port 389 as it is in installations where CA is by default?

Can LDAPS be used only for spawning the CA db on first master and then migrated to TLS so that installations of additional replicas can use TLS?

Comment 25 Endi Sukma Dewata 2016-08-05 20:49:52 UTC
I'd suggest we investigate the second issue (i.e. LDAPS issue) after we fix the first one (i.e. missing 3rd-party certs in PKCS #12 file). I need to investigate the proper procedure to set up a CA replica when the CA master is configured with LDAPS.

Comment 26 Endi Sukma Dewata 2016-08-09 18:43:41 UTC
Hi Petr,

Do you know which SSL certificate is supposed to be used by the HTTP proxy in this scenario (see comment #19)? Thanks.

Comment 27 Florence Blanc-Renaud 2016-08-10 13:49:20 UTC
Hi Endi,

as IPA was initially CA_less, HTTP is using a 3rd part certificate. When ipa-ca-install is run on the master, the cert for HTTP is not modified and is still the one signed by the 3rd part CA.
I must check how we can provide this trust anchor to pkispawn.

Comment 28 Petr Vobornik 2016-08-12 13:36:40 UTC
Upstream ticket:
https://fedorahosted.org/freeipa/ticket/6207

Comment 29 Martin Bašti 2016-08-26 14:33:09 UTC
Fixed upstream
master:
https://fedorahosted.org/freeipa/changeset/6581389ac3ac1c6a0dbeb18d80e3fef69b158cc8


bz1365858 must be fixed too for sucessful installation

Comment 30 Martin Bašti 2016-09-01 11:13:19 UTC
Fixed upstream
master:
https://fedorahosted.org/freeipa/changeset/17ea4ae6b9007e121ae1ea7748643394fec84ad7

Comment 33 Abhijeet Kasurde 2016-09-19 11:28:46 UTC
Marking this BZ as FailedQA, because installer is not successful and bz1365858 must be fixed for successful installation.

Comment 42 Michal Reznik 2017-05-16 11:27:32 UTC
FAILED_QA

Hit an issue which might be related to:

https://pagure.io/freeipa/issue/6577

ipa-server-4.5.0-11.el7.x86_64
pki-ca-10.4.1-4.el7.noarch

1. Install IPA server 

[root@master ~]# ipa-server-install --ip-address $(ip addr|grep "global"|cut -d " " -f6|cut -d "/" -f1|head -n 1) -r TESTRELM.TEST -n testrelm.test -p 'Secret123' -a 'Secret123' --setup-dns --forwarder 192.168.222.1 -U --dirsrv-cert-file=./server.p12 --http-cert-file=./server.p12 --dirsrv-pin Secret123 --http-pin Secret123 --no-pkinit

2. Install CA-less replica 

[root@replica ~]# ipa-client-install -U --domain testrelm.test --realm TESTRELM.TEST -p admin -w Secret123 --server master.testrelm.test

[root@replica ~]# ipa-replica-install -U --dirsrv-cert-file=./replica.p12 --http-cert-file=./replica.p12 --dirsrv-pin Secret123 --http-pin Secret123 -P admin -w Secret123 --no-pkinit

3. Promote IPA server from CA-less to CA-full

[root@master ~]# ipa-ca-install

4. Promote IPA replica from CA-less to CA-full

[root@replica1 ~]# ipa-ca-install
Directory Manager (existing master) password: 

Run connection check to master
Connection check OK
/usr/lib/python2.7/site-packages/urllib3/connection.py:251: SecurityWarning: Certificate has no `subjectAltName`, falling back to check for a `commonName` for now. This feature is being removed by major browsers and deprecated by RFC 2818. (See https://github.com/shazow/urllib3/issues/497 for details.)
  SecurityWarning
Configuring certificate server (pki-tomcatd). Estimated time: 3 minutes
  [1/27]: creating certificate server db
  [2/27]: setting up initial replication
Starting replication, please wait until this has completed.
Update in progress, 3 seconds elapsed
Update succeeded

    [3/27]: creating installation admin user
  [4/27]: configuring certificate server instance
  [5/27]: exporting Dogtag certificate store pin
  [6/27]: stopping certificate server instance to update CS.cfg
  [7/27]: backing up CS.cfg
  [8/27]: disabling nonces
  [9/27]: set up CRL publishing
  [10/27]: enable PKIX certificate path discovery and validation
  [11/27]: destroying installation admin user
  [12/27]: starting certificate server instance
  [13/27]: configure certmonger for renewals
  [14/27]: Importing RA key
  [15/27]: setting up signing cert profile
  [16/27]: setting audit signing renewal to 2 years
  [17/27]: restarting certificate server
  [18/27]: authorizing RA to modify profiles
  [19/27]: authorizing RA to manage lightweight CAs
  [20/27]: Ensure lightweight CAs container exists
  [21/27]: configure certificate renewals
  [22/27]: configure Server-Cert certificate renewal
  [23/27]: Configure HTTP to proxy connections
  [24/27]: restarting certificate server
  [25/27]: updating IPA configuration
  [26/27]: enabling CA instance
  [27/27]: configuring certmonger renewal for lightweight CAs
Done configuring certificate server (pki-tomcatd).

Your system may be partly configured.
Run /usr/sbin/ipa-server-install --uninstall to clean up.

CA did not start in 300.0s

[root@replica1 ~]# ipactl status
Directory Service: RUNNING
krb5kdc Service: RUNNING
kadmin Service: RUNNING
httpd Service: RUNNING
ipa-custodia Service: RUNNING
ntpd Service: RUNNING
pki-tomcatd Service: STOPPED
ipa-otpd Service: RUNNING

[root@replica1 ~]# systemctl status pki-tomcatd@pki-tomcat.servicepki-tomcatd@pki-tomcat.service - PKI Tomcat Server pki-tomcat
   Loaded: loaded (/usr/lib/systemd/system/pki-tomcatd@.service; enabled; vendor preset: disabled)
   Active: active (running) since Tue 2017-05-16 07:02:14 EDT; 8min ago
  Process: 22844 ExecStartPre=/usr/bin/pkidaemon start %i (code=exited, status=0/SUCCESS)
 Main PID: 22970 (java)
   CGroup: /system.slice/system-pki\x2dtomcatd.slice/pki-tomcatd@pki-tomcat.service
           └─22970 /usr/lib/jvm/jre-1.8.0-openjdk/bin/java -DRESTEASY_LIB=/usr/share/java/resteasy-base -Djava.library.path=/usr/lib64/nuxwdog-jni -classpath /usr/share/tomcat/bin/bootstrap.jar:/usr/share/tom...

May 16 07:10:19 replica1 server[22970]: WARNING: Exception processing realm com.netscape.cms.tomcat.ProxyRealm@5c47a92a background process
May 16 07:10:19 replica1 server[22970]: javax.ws.rs.ServiceUnavailableException: Subsystem unavailable
May 16 07:10:19 replica1 server[22970]: at com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137)
May 16 07:10:19 replica1 server[22970]: at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1356)
May 16 07:10:19 replica1 server[22970]: at org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5958)
May 16 07:10:19 replica1 server[22970]: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1542)
May 16 07:10:19 replica1 server[22970]: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552)
May 16 07:10:19 replica1 server[22970]: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552)
May 16 07:10:19 replica1 server[22970]: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1520)
May 16 07:10:19 replica1 server[22970]: at java.lang.Thread.run(Thread.java:748)

Comment 43 Michal Reznik 2017-05-16 11:29:38 UTC
Created attachment 1279275 [details]
pki-ca-spawn

Comment 44 Michal Reznik 2017-05-16 11:30:47 UTC
Created attachment 1279276 [details]
ca_debug20170516

Comment 45 Michal Reznik 2017-05-16 11:31:59 UTC
Created attachment 1279277 [details]
journalctl_pki-tomcat20170516

Comment 46 Michal Reznik 2017-05-16 12:52:13 UTC
Additional info:

[root@master ~]# ipa cert-find
ipa: ERROR: cannot connect to 'https://master.testrelm.test/ipa/json': [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:579)

[root@master ~]# python /usr/lib/python2.7/site-packages/requests/certs.py
/etc/pki/tls/certs/ca-bundle.crt

[root@master ~]# openssl crl2pkcs7 -nocrl -certfile /etc/pki/tls/certs/ca-bundle.crt | openssl pkcs7 -print_certs -noout | grep -i example
subject=/O=Example Organization/CN=CA
issuer=/O=Example Organization/CN=CA

[root@replica1 ~]# ipa cert-find
ipa: ERROR: an internal error has occurred

Comment 47 Kaleem 2017-05-16 14:41:37 UTC
Marking failed_qa as per comments 42-46

Comment 48 Michal Reznik 2017-05-17 10:31:07 UTC
Temporarily withdrawing "failed_qa" as the error might have been caused by bad certificates. Will need to run the checks again.

Comment 49 Michal Reznik 2017-05-17 11:39:55 UTC
UPDATE: there were no issue with certificates as I originally thought.

The issue seems to be the same as originally reported with pki-server-10.4.1-3.el7.noarch.

  [3/27]: creating installation admin user
  [4/27]: configuring certificate server instance
ipa.ipaserver.install.cainstance.CAInstance: CRITICAL Failed to configure CA instance: Command '/usr/sbin/pkispawn -s CA -f /tmp/tmpzMUgcv' returned non-zero exit status 1
ipa.ipaserver.install.cainstance.CAInstance: CRITICAL See the installation logs and the following files/directories for more information:
ipa.ipaserver.install.cainstance.CAInstance: CRITICAL   /var/log/pki/pki-tomcat
  [error] RuntimeError: CA configuration failed.

After upgrade to pki-server-10.4.1-4.el7.noarch we are hitting the issue as per comments 42-46. Attaching new logs.

Comment 50 Michal Reznik 2017-05-17 11:41:28 UTC
Created attachment 1279654 [details]
20170517_master_journaclt_pki-tomcat

Comment 51 Michal Reznik 2017-05-17 11:42:27 UTC
Created attachment 1279655 [details]
20170517_master_ca-debug

Comment 52 Michal Reznik 2017-05-17 11:43:16 UTC
Created attachment 1279656 [details]
20170517_master_pki-spawn071437

Comment 53 Michal Reznik 2017-05-17 11:44:27 UTC
Created attachment 1279657 [details]
20170517_replica_journaclt_pki-tomcat

Comment 54 Michal Reznik 2017-05-17 11:45:51 UTC
Created attachment 1279659 [details]
20170517_replica_pki-spawn071846

Comment 55 Michal Reznik 2017-05-17 11:47:25 UTC
Created attachment 1279660 [details]
20170517_replica_ca-debug

Comment 56 Florence Blanc-Renaud 2017-05-17 14:53:17 UTC
Hi Michal,

the procedure in comment 42 is not complete as it lacks the step running ipa-certupdate on all machines after ipa-ca-install.
If this step is not run, then the replica does not download the new CA cert, and when Dogtag starts on the replica, it fails to connect to LDAP server (it connects through SSL port using subsystemCert, which is delivered by the new CA, and LDAP server does not trust this CA until ipa-certupdate has been run).

This does not explain why ipa-ca-install fails on the master, though. I am still investigating the issue.

Comment 57 Florence Blanc-Renaud 2017-05-18 10:01:39 UTC
Hi Michal,

please ignore the last line of my comment 56:
"This does not explain why ipa-ca-install fails on the master, though. I am still investigating the issue."
I wrongly assumed that the error happened during ipa-ca-install on the master. If the error happens on the replica, then it is expected as the ipa-certupdate step is missing. But the failure should happen when pki-tomcat is restarted after step 27, not during pkispawn.

The attached logs do not show any issue during pkispawn, can you check if they are the right ones, and also try the procedure with the ipa-certupdate step on the replica after ipa-ca-install on the master?
Thank you

Comment 58 Michal Reznik 2017-05-19 09:53:02 UTC
Hi Flo,

I was able to issue "ipa-certupdate" on both master and replica after [SSL: CERTIFICATE_VERIFY_FAILED] issue workaround (adding CA cert that signed the httpd cert to /etc/ipa/ca.crt).

Then with pki-server-10.4.1-4.el7.noarch I was able to successfully proceed with all steps. Thank you for your assistance.

Comment 59 Michal Reznik 2017-05-19 11:54:25 UTC
Verified on:

ipa-server-4.5.0-13.el7.x86_64
pki-server-10.4.1-4.el7.noarch
selinux-policy-3.13.1-150.el7.noarch

[root@master ~]# getenforce
Enforcing

[root@replica ~]# getenforce
Enforcing

1. Install CA-less IPA server 

[root@master ~]# ipa-server-install --ip-address $(ip addr|grep "global"|cut -d " " -f6|cut -d "/" -f1|head -n 1) -r TESTRELM.TEST -n testrelm.test -p 'XXX' -a 'XXX' --setup-dns --forwarder 192.168.222.1 -U --dirsrv-cert-file=./server.p12 --http-cert-file=./server.p12 --dirsrv-pin XXX --http-pin XXX --no-pkinit

2. Install CA-less replica 

[root@replica ~]# ipa-client-install -U --domain testrelm.test --realm TESTRELM.TEST -p admin -w XXX --server master.testrelm.test
[root@replica ~]# ipa-replica-install -U --dirsrv-cert-file=./replica.p12 --http-cert-file=./replica.p12 --dirsrv-pin XXX --http-pin XXX -P admin -w XXX --no-pkinit

3. Promote IPA server from CA-less to CA-full

[root@master ~]# ipa-ca-install

Workaround for [SSL: CERTIFICATE_VERIFY_FAILED] (not related to the tested issue):

[root@master2 ~]# cat nssdb/ca1.pem >> /etc/ipa/ca.crt

4. Update local IPA certificate databases with certificates from the server

[root@master ~]# ipa-certupdate
trying https://master.testrelm.test/ipa/json
Forwarding 'ca_is_enabled' to json server 'https://master.testrelm.test/ipa/json'
Forwarding 'ca_find/1' to json server 'https://master.testrelm.test/ipa/json'
Systemwide CA database updated.
Systemwide CA database updated.
The ipa-certupdate command was successful

5. Update local IPA certificate databases with certificates from the server

[root@replica ~]# kinit admin
Password for admin@TESTRELM.TEST: 
[root@replica ~]# 

[root@replica ~]# ipa-certupdate
trying https://replica.testrelm.test/ipa/json
Forwarding 'ca_is_enabled' to json server 'https://replica.testrelm.test/ipa/json'
Forwarding 'ca_find/1' to json server 'https://replica.testrelm.test/ipa/json'
Systemwide CA database updated.
Systemwide CA database updated.
The ipa-certupdate command was successful

6. Promote IPA replica from CA-less to CA-full

[root@replica ~]# ipa-ca-install
Directory Manager (existing master) password: 

Run connection check to master
Connection check OK
/usr/lib/python2.7/site-packages/urllib3/connection.py:251: SecurityWarning: Certificate has no `subjectAltName`, falling back to check for a `commonName` for now. This feature is being removed by major browsers and deprecated by RFC 2818. (See https://github.com/shazow/urllib3/issues/497 for details.)
  SecurityWarning
Configuring certificate server (pki-tomcatd). Estimated time: 3 minutes
  [1/27]: creating certificate server db
  [2/27]: setting up initial replication
Starting replication, please wait until this has completed.
Update in progress, 4 seconds elapsed
Update succeeded

  [3/27]: creating installation admin user
  [4/27]: configuring certificate server instance
  [5/27]: exporting Dogtag certificate store pin
  [6/27]: stopping certificate server instance to update CS.cfg
  [7/27]: backing up CS.cfg
  [8/27]: disabling nonces
  [9/27]: set up CRL publishing
  [10/27]: enable PKIX certificate path discovery and validation
  [11/27]: destroying installation admin user
  [12/27]: starting certificate server instance
  [13/27]: configure certmonger for renewals
  [14/27]: Importing RA key
  [15/27]: setting up signing cert profile
  [16/27]: setting audit signing renewal to 2 years
  [17/27]: restarting certificate server
  [18/27]: authorizing RA to modify profiles
  [19/27]: authorizing RA to manage lightweight CAs
  [20/27]: Ensure lightweight CAs container exists
  [21/27]: configure certificate renewals
  [22/27]: configure Server-Cert certificate renewal
  [23/27]: Configure HTTP to proxy connections
  [24/27]: restarting certificate server
  [25/27]: updating IPA configuration
  [26/27]: enabling CA instance
  [27/27]: configuring certmonger renewal for lightweight CAs
Done configuring certificate server (pki-tomcatd).
Updating DNS system records

Comment 60 errata-xmlrpc 2017-08-01 09:39:54 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:2304


Note You need to log in before you can comment on or make changes to this bug.