Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 1354070 - DNS service named in one of our IPA server cannot start
Summary: DNS service named in one of our IPA server cannot start
Keywords:
Status: CLOSED INSUFFICIENT_DATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: ipa
Version: 7.2
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: rc
: ---
Assignee: IPA Maintainers
QA Contact: Kaleem
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-07-09 00:51 UTC by lmgnid
Modified: 2016-08-12 10:50 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2016-08-12 10:50:55 UTC


Attachments (Terms of Use)

Description lmgnid 2016-07-09 00:51:27 UTC
Description of problem:
In one of our IPA server, named service suddenly cannot start:
Job for named-pkcs11.service failed because the control process exited with erro-pkcs11.service" and "journalctl -xe" for details.


Version-Release number of selected component (if applicable):
[root@eupreprd-ops-ipa-01 ~]# rpm -qa | grep ipa
python-iniparse-0.4-9.el7.noarch
libipa_hbac-1.13.0-40.el7_2.4.x86_64
sssd-ipa-1.13.0-40.el7_2.4.x86_64
ipa-server-4.2.0-15.el7_2.15.x86_64
python-libipa_hbac-1.13.0-40.el7_2.4.x86_64
redhat-access-plugin-ipa-0.9.1-2.el7.noarch
ipa-client-4.2.0-15.el7_2.15.x86_64
ipa-server-dns-4.2.0-15.el7_2.15.x86_64
ipa-python-4.2.0-15.el7_2.15.x86_64
ipa-admintools-4.2.0-15.el7_2.15.x86_64


How reproducible:
Everytime


Steps to Reproduce:
1.ipactl start


Actual results:
named cannot start


Expected results:
named should start


Additional info:
In one of our IPA server, named service suddenly cannot start, so I followed  the link bellow:
https://fedorahosted.org/bind-dyndb-ldap/wiki/BIND9/NamedCannotStart

Found some errors like bellow:
==> messages <==
Jul  8 23:30:30 eupreprd-ops-ipa-01 named-pkcs11[5002]: LDAP error: Invalid credentials: SASL(-14): authorization failure: : bind to LDAP server failed

It should be a “Invalid credentials: bind to LDAP server failed “ error, however, the commands bellow shows no issues to me:
[root@eupreprd-ops-ipa-01 ~]# kvno DNS/eupreprd-ops-ipa-01.internal.com@INTERNAL.COM
DNS/eupreprd-ops-ipa-01.internal.com@INTERNAL.COM: kvno = 2
[root@eupreprd-ops-ipa-01 ~]# klist -kt /etc/named.keytab
Keytab name: FILE:/etc/named.keytab
KVNO Timestamp           Principal
---- ------------------- ------------------------------------------------------
   2 06/10/2016 17:57:38 DNS/eupreprd-ops-ipa-01.internal.com@INTERNAL.COM
   2 06/10/2016 17:57:38 DNS/eupreprd-ops-ipa-01.internal.com@INTERNAL.COM
   2 06/10/2016 17:57:38 DNS/eupreprd-ops-ipa-01.internal.com@INTERNAL.COM
   2 06/10/2016 17:57:38 DNS/eupreprd-ops-ipa-01.internal.com@INTERNAL.COM
   2 06/10/2016 17:57:38 DNS/eupreprd-ops-ipa-01.internal.com@INTERNAL.COM
   2 06/10/2016 17:57:38 DNS/eupreprd-ops-ipa-01.internal.com@INTERNAL.COM

[root@eupreprd-ops-ipa-01 ~]# kinit -kt /etc/named.keytab DNS/eupreprd-ops-ipa-01.internal.com
[root@eupreprd-ops-ipa-01 ~]

[root@eupreprd-ops-ipa-01 ~]# ldapsearch -H 'ldapi://%2fvar%2frun%2fslapd-INTERNAL-COM.socket"' -Y GSSAPI -b 'cn=dns, dc=internal,dc=com'
…<Lots of results, will not put here>…

For now, I have use the “(Workaround) Use simple LDAP BIND insted of Kerberos” to make it work, but still want to know how to recover to “sasl”? 

Thanks in advance!

Comment 2 Petr Spacek 2016-07-12 11:54:08 UTC
Please be so kind and continue with discussion on
https://www.redhat.com/archives/freeipa-users/2016-July/msg00153.html

Bugzilla is not a support tool. We will open/extend the bug at the point where it is clear that this is really a bug and what caused it.

Thank you for understanding.

Comment 3 lmgnid 2016-07-13 23:44:53 UTC
Hi Petr,

Thanks for you reply, but I didn't find a to comment in your link. So I have to still use this post here:

When I tried today, I got the "authorization" error, so I tried to renew the named.keytab, I always got this "PrincipalName not found" even the "PrinciplaName" is there, do you have any ideas?


[root@eupreprd-ops-ipa-01 ~]# kinit -kt /etc/named.keytab DNS/eupreprd-ops-ipa-01.internal.com
[root@eupreprd-ops-ipa-01 ~]# ldapsearch -Y GSSAPI -h `hostname` -b "" -s base
SASL/GSSAPI authentication started
ldap_sasl_interactive_bind_s: Invalid credentials (49)
        additional info: SASL(-14): authorization failure:

[root@eupreprd-ops-ipa-01 slapd-INTERNAL-COM]# ipa service-show DNS/eupreprd-ops-ipa-01.internal.com
  Principal: DNS/eupreprd-ops-ipa-01.internal.com@INTERNAL.COM
  Keytab: True
  Managed by: eupreprd-ops-ipa-01.internal.com

[root@eupreprd-ops-ipa-01 slapd-INTERNAL-COM]# ipa-getkeytab -s eupreprd-ops-ipa-01 -p dns/eupreprd-ops-ipa-01.internal.com@INTERNAL.COM -k /tmp/named.keytab.new
Failed to parse result: PrincipalName not found.

Retrying with pre-4.0 keytab retrieval method...
Failed to parse result: PrincipalName not found.

Failed to get keytab!
Failed to get keytab

[root@eupreprd-ops-ipa-01 slapd-INTERNAL-COM]# ipa-getkeytab -s usqa-ops-ipa-02 -p dns/eupreprd-ops-ipa-01.internal.com@INTERNAL.COM -k /tmp/named.keytab.new
Failed to parse result: PrincipalName not found.

Retrying with pre-4.0 keytab retrieval method...
Failed to parse result: PrincipalName not found.

Failed to get keytab!
Failed to get keytab

Comment 4 Petr Vobornik 2016-07-15 13:47:21 UTC
There are different casing in the examples:
* dns/eupreprd-ops-ipa-01.internal.com@INTERNAL.COM
* DNS/eupreprd-ops-ipa-01.internal.com@INTERNAL.COM

make sure to use the same

Comment 5 lmgnid 2016-07-15 17:26:37 UTC
Hi Petr,

Thanks for your comment, I tried and still the same:

[root@eupreprd-ops-ipa-01 tmp]# kinit admin
Password for admin@INTERNAL.COM:
[root@eupreprd-ops-ipa-01 tmp]# ipa-getkeytab -s eupreprd-ops-ipa-01 -p DNS/eupreprd-ops-ipa-01.internal.com@INTERNAL.COM -k /tmp/named.keytab.new
Failed to parse result: PrincipalName not found.

Retrying with pre-4.0 keytab retrieval method...
Failed to parse result: PrincipalName not found.

Failed to get keytab!
Failed to get keytab
[root@eupreprd-ops-ipa-01 tmp]# ipa-getkeytab -s usqa-ops-ipa-02 -p DNS/eupreprd-ops-ipa-01.internal.com@INTERNAL.COM -k /tmp/named.keytab.new
Failed to parse result: PrincipalName not found.

Retrying with pre-4.0 keytab retrieval method...
Failed to parse result: PrincipalName not found.

Failed to get keytab!
Failed to get keytab

Comment 6 Petr Vobornik 2016-07-18 08:04:06 UTC
what is the output of following `ipa service-show` command?
$ kinit admin
$ ipa service-show DNS/eupreprd-ops-ipa-01.internal.com@INTERNAL.COM --all

Comment 7 lmgnid 2016-07-18 17:25:54 UTC
Here you are:

[root@eupreprd-ops-ipa-01 ~]# ipa service-show DNS/eupreprd-ops-ipa-01.internal.com@INTERNAL.COM --all
  dn: krbprincipalname=DNS/eupreprd-ops-ipa-01.internal.com@INTERNAL.COM,cn=services,cn=accounts,dc=internal,dc=com
  Principal: DNS/eupreprd-ops-ipa-01.internal.com@INTERNAL.COM
  Requires pre-authentication: True
  Trusted for delegation: False
  Keytab: True
  Managed by: eupreprd-ops-ipa-01.internal.com
  ipakrbprincipalalias: DNS/eupreprd-ops-ipa-01.internal.com@INTERNAL.COM
  ipauniqueid: d1e8811e-2f34-11e6-b4d4-02590ce2d33f
  krbextradata: AAKS/1pXcm9vdC9hZG1pbkBJTlRFUk5BTC5TQU1TVU5HS05PWC5DT00A
  krblastpwdchange: 20160610175738Z
  krblastsuccessfulauth: 20160715010544Z
  krbloginfailedcount: 0
  objectclass: ipaobject, top, ipaservice, pkiuser, ipakrbprincipal, krbprincipal, krbprincipalaux,
               krbTicketPolicyAux

Comment 8 Petr Spacek 2016-07-21 08:45:10 UTC
Interesting. This might be some replication problem or so. Please follow http://www.freeipa.org/page/Troubleshooting#Replication_issues and check it replication works.

BTW I would recommend you to check is command ipa service-show DNS/eupreprd-ops-ipa-01.internal.com@INTERNAL.COM --all returns the very same result on all IPA servers in the topology.

Comment 9 Petr Vobornik 2016-07-21 16:07:39 UTC
small note, not to be surprised:
   ipa-getkeytab by default creates new key on a server which will invalidate any other already downloaded keys. This is usually OK since there should be ideally only one copy of a keytab but it might surprise during tested.

Add -r option to prevent regeneration of new key.

Comment 10 lmgnid 2016-07-22 00:34:19 UTC
@Petr, Yes I Do see the replication errors, similar to DNS credential error. And it seems different server shows different records. But as I cannot update the key with "ipa-getkeytab", do you have any ideas for how to solve this "Invalid Credential" error for replication and DNS? Thanks!

[root@eupreprd-ops-ipa-01 tmp]# tail -f /var/log/dirsrv/slapd-INTERNAL-COM/errors
[15/Jul/2016:16:42:06 +0000] slapd_ldap_sasl_interactive_bind - Error: could not perform interactive bind for id [] mech [GSSAPI]: LDAP error 49 (Invalid credentials) (SASL(-14): authorization failure: security flags do not match required) errno 0 (Success)
[15/Jul/2016:16:42:06 +0000] slapi_ldap_bind - Error: could not perform interactive bind for id [] authentication mechanism [GSSAPI]: error 49 (Invalid credentials)
[15/Jul/2016:16:47:05 +0000] slapd_ldap_sasl_interactive_bind - Error: could not perform interactive bind for id [] mech [GSSAPI]: LDAP error 49 (Invalid credentials) (SASL(-14): authorization failure: ) errno 0 (Success)
[15/Jul/2016:16:47:05 +0000] slapi_ldap_bind - Error: could not perform interactive bind for id [] authentication mechanism [GSSAPI]: error 49 (Invalid credentials)
[15/Jul/2016:16:47:05 +0000] slapd_ldap_sasl_interactive_bind - Error: could not perform interactive bind for id [] mech [GSSAPI]: LDAP error 49 (Invalid credentials) (SASL(-14): authorization failure: security flags do not match required) errno 0 (Success)


[root@usqa-ops-ipa-02 ~]# ipa service-show DNS/eupreprd-ops-ipa-01.internal.com@INTERNAL.COM --all
  dn: krbprincipalname=DNS/eupreprd-ops-ipa-01.internal.com@INTERNAL.COM,cn=services,cn=accounts,dc=internal,dc=com
  Principal: DNS/eupreprd-ops-
             ipa-01.internal.com@INTERNAL.COM
  Requires pre-authentication: True
  Trusted for delegation: False
  Keytab: True
  Managed by: eupreprd-ops-ipa-01.internal.com
  ipakrbprincipalalias: DNS/eupreprd-ops-
                        ipa-01.internal.com@INTERNAL.COM
  ipauniqueid: 36d738d4-b665-11e5-af0a-02590ce2d33f
  krbextradata: AAKnT5BWcm9vdC9hZG1pbkBJTlRFUk5BTC5TQU1TVU5HS05PWC5DT00A
  krblastpwdchange: 20160109000911Z
  objectclass: ipaobject, top, ipaservice, pkiuser, ipakrbprincipal,
               krbprincipal, krbprincipalaux, krbTicketPolicyAux


[root@eupreprd-ops-ipa-01 ~]# ipa service-show DNS/eupreprd-ops-ipa-01.internal.com@INTERNAL.COM --all
  dn: krbprincipalname=DNS/eupreprd-ops-ipa-01.internal.com@INTERNAL.COM,cn=services,cn=accounts,dc=internal,dc=com
  Principal: DNS/eupreprd-ops-
             ipa-01.internal.com@INTERNAL.COM
  Requires pre-authentication: True
  Trusted for delegation: False
  Keytab: True
  Managed by: eupreprd-ops-ipa-01.internal.com
  ipakrbprincipalalias: DNS/eupreprd-ops-
                        ipa-01.internal.com@INTERNAL.COM
  ipauniqueid: d1e8811e-2f34-11e6-b4d4-02590ce2d33f
  krbextradata: AAKS/1pXcm9vdC9hZG1pbkBJTlRFUk5BTC5TQU1TVU5HS05PWC5DT00A
  krblastpwdchange: 20160610175738Z
  krblastsuccessfulauth: 20160715010544Z
  krbloginfailedcount: 0
  objectclass: ipaobject, top, ipaservice, pkiuser, ipakrbprincipal,
               krbprincipal, krbprincipalaux, krbTicketPolicyAux

Comment 11 Petr Spacek 2016-07-22 10:16:19 UTC
Well, the first step is to fix replication. After that the ipa-getkeytab trick should just work.

Please join freeipa-users mailing list and continue with discussion there, Bugzilla is really not suitable as support tool.

Form for joining mailing list can be found at:
https://www.redhat.com/mailman/listinfo/freeipa-users

Comment 12 Martin Bašti 2016-07-27 11:46:16 UTC
Looks like a replication issue, not enough data or reproducer provided, closing BZ.

Feel free to reopen this BZ if your problem persists or happen again.

Comment 13 lmgnid 2016-07-27 17:02:27 UTC
Hello, this issue is still there, I tried everything from the redhat guides or mails list posts, but still the same. If you need more data or information, please let me know.

Comment 14 Petr Spacek 2016-08-12 10:50:55 UTC
I'm not really sure what to suggest because you did not say what you have tried, what output from commands you got etc.

I would suggest you to do following:
1. Joining mailing list freeipa-users:
https://www.redhat.com/mailman/listinfo/freeipa-users

2. Write an e-mail with all the information as explained in
http://www.chiark.greenend.org.uk/~sgtatham/bugs.html

3. We can re-open this bug at the moment when the root cause is known. Bugzilla is not a support tool, we need to capture only the root cause here. Before the root cause is know, please be so kind and use mailing list.

Thank you for understanding!


Note You need to log in before you can comment on or make changes to this bug.