Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 1063732 - pam_krb5 memory leaks
Summary: pam_krb5 memory leaks
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: krb5
Version: 7.0
Hardware: All
OS: Linux
high
high
Target Milestone: rc
: ---
Assignee: Nalin Dahyabhai
QA Contact: Patrik Kis
URL:
Whiteboard:
Depends On: 643962
Blocks:
TreeView+ depends on / blocked
 
Reported: 2014-02-11 10:54 UTC by Patrik Kis
Modified: 2014-06-18 01:08 UTC (History)
13 users (show)

Fixed In Version: krb5-1.11.3-49.el7
Doc Type: Bug Fix
Doc Text:
Clone Of: 643962
: 1075647 (view as bug list)
Environment:
Last Closed: 2014-06-13 09:59:28 UTC


Attachments (Terms of Use)

Description Patrik Kis 2014-02-11 10:54:11 UTC
It looks pam_krb5 leaks the memory. The test below fails on the latest rhel-7 compose with:

pam_krb5-2.4.8-3.el7
krb5-1.11.3-45.el7

While on rhel6 the memory consumption does not increase at all regardless how long the test program is running, on rhel7 the memory consumption increases significantly after 64 iteration:

x86_64 increase by 33%
ppc64  increase by more than 100%

This is from ppc64 system:
# ./pamAuthTest.o krbuser krbuserpass 64
Authenticated

 total             4992K
Authenticated

 total             4992K
Authenticated

 total             5184K

... SNIP ...

Authenticated

 total            11136K
Authenticated

 total            11200K
Authenticated

 total            11328K
Authenticated

 total            11456K
Authenticated

 total            11456K

All the test details can be found below and in the original bug. I run the test against pam password-auth, so no pam_krb5 is called, and it did not leak. Should be anything unclear about the test feel free to contact me; I have a test machine also with the reproducer.


+++ This bug was initially created as a clone of Bug #643962 +++

Description of problem:
pam_authenticate leaks memory with every call for users on Active Directory. I have attached the source code of the program with which problem was replicated (See pam_auth_test.c attached with this ticket). 

Version-Release number of selected component (if applicable):
RHEL 5.4 64-bit
pam-0.99.6.2-6.el5

How reproducible:
Configure Active Directory on the system with winbind and run the sample program attached with this ticket.

How to run the program:
1.	Copy runAuthTest.sh, pamAuthTest.o, system-auth,  check_user to the system. (Please find the files attached with this ticket).
2.      Copy system-auth and check_user file to /etc/pam.d/ directory.
4.	Edit runAuthTest.sh to specify username and password of an AD user.
5.	Run ./runAuthTest.sh
6.	Output is written to /var/log/authtest.log by default.

We can see that the process memory size grows.

Environment:
-------------
RHEL 5.4 64-bit
SAMBA version 3
Winbind Version 3.5.6-43
AD – Windows 2008 server AD

PAM libs installed in customer environment (RHEL 5u4)
----------------------------
pam_smb-1.1.7-7.2.1
pam_ccreds-3-5
pam_smb-1.1.7-7.2.1
pam_passwdqc-1.0.2-1.2.2
pam-0.99.6.2-6.el5
pam_krb5-2.2.14-10
pam_ccreds-3-5
pam_pkcs11-0.5.3-23
pam_krb5-2.2.14-10
pam_passwdqc-1.0.2-1.2.2
pam-0.99.6.2-6.el5
pam_pkcs11-0.5.3-23

PAM Configuration
------------------------
/etc/pam.d/check_user
#%PAM-1.0
auth        required    pam_stack.so service=system-auth
account     required    pam_stack.so service=system-auth
password    required    pam_stack.so service=system-auth
session     required    pam_stack.so service=system-auth


/etc/pam.d/system-auth
auth        required      pam_env.so
auth        sufficient    pam_unix.so nullok try_first_pass
auth        requisite     pam_succeed_if.so uid >= 500 quiet
auth        sufficient    pam_krb5.so use_first_pass
auth        required      pam_deny.so
account     required      pam_access.so
account     required      pam_unix.so broken_shadow
account     sufficient    pam_succeed_if.so uid < 500 quiet
account     [default=bad success=ok user_unknown=ignore] pam_krb5.so
account     required      pam_permit.so
password    requisite     pam_cracklib.so try_first_pass retry=3
password    sufficient    pam_unix.so md5 shadow nullok try_first_pass use_authtok
password    sufficient    pam_krb5.so use_authtok
password    required      pam_deny.so
session     optional      pam_keyinit.so revoke
session     required      pam_limits.so
session     optional      pam_oddjob_mkhomedir.so
session     [success=1 default=ignore] pam_succeed_if.so service in crond quiet use_uid
session     required      pam_unix.so
session     optional      pam_krb5.so
-------------------------------


Steps to Reproduce:
1. Setup Active Directory, Samba and winbindd on the system.
2. Define /etc/pam.d/system-auth file as specified above.
3.    runAuthTest.sh, pamAuthTest.o, system-auth and check_user to the system. (See attached with this ticket).
4.    Copy system-auth and check_user file to /etc/pam.d/ directory.
5.    Edit runAuthTest.sh to specify username and password of an Active Directory user.
6.    Run runAuthTest.sh
7.    Program writes process memory size to /var/log/authtest.log by default.

We can see that process memory size grows and its memory size reported in the log.

Actual results:
Memory leaks with every pam_authenticate call.

Expected results:
Memory shouldn't leak.

Additional info:
1. Linux kickstart file is attached. (Please see attached ks.cfg)
2. /var/log/messages attached (Please see attached messages.zip) where we see warning messages from winbindd.
3. List of packages installed on the system. (Please see attached packages)
4. Test program to reproduce the issue:
runAuthTest.sh - script to call the program
pamAuthTest.o - Calls pam_authenticate.
And its source file are pam_auth_test.c and Utils.c
check_user and system-auth - PAM configuration file.

--- Additional comment from Tomas Mraz on 2010-10-18 19:48:53 CEST ---

Can you please run the exactly same test against a local user? Does it still leak?

--- Additional comment from  on 2010-10-19 06:02:52 CEST ---

Yes, it is not leaking against local user.

--- Additional comment from Nalin Dahyabhai on 2010-10-20 18:06:05 CEST ---

I think there's probably a leak somewhere, most of it lost to matchpathcon() initializing its contexts every time it's loaded as a dependency of libkrb5, which is a dependency of pam_krb5, and losing track of the memory when it's subsequently unloaded because there's no way for libkrb5 to be sure it's safe to clean that memory up.

Switching to using the newer APIs available in libselinux 2.x should make it possible to avoid this loss, but in the meantime, linking the calling application directly with libselinux (or causing it to be loaded by setting $LD_PRELOAD -- anything to keep the library refcount above 0 at all times so that it won't actually be unloaded) could work around it.

Comment 2 Patrik Kis 2014-02-11 14:00:56 UTC
I did some more investigation and found that the trigger for this leak is krb5-pkinit package. If it is not installed the leak does not appear. I leave the component on pam_krb5 for now, if the issue really is in krb5-pkinit, please change it.

Comment 3 Patrik Kis 2014-02-12 09:08:08 UTC
Actually the problem appears also on RHEL-6 when krb5-pkinit-openssl is installed so this is probably not a regression.

Comment 4 Nalin Dahyabhai 2014-03-11 20:02:30 UTC
The majority of the leaks are happening when libkrb5, which is pulled in and out along with pam_krb5, loads the pkinit module, initializes libcrypto, and then unloads the pkinit module.  The memory allocated during libcrypto initialization is lost, so the next time we try to check a password, it's allocated again.

The bad news is that because pam_krb5 isn't loading pkinit directly, it's not in a position to do anything to keep it from being unloaded for this case, as it doesn't have a direct dependency on the plugin.  The most expedient thing is to tag the pkinit module itself with RTLD_NODELETE, accept the now one-time allocation in libcrypto, and continue to chase the leaks which take longer to pile up.

Comment 7 Ludek Smid 2014-06-13 09:59:28 UTC
This request was resolved in Red Hat Enterprise Linux 7.0.

Contact your manager or support representative in case you have further questions about the request.


Note You need to log in before you can comment on or make changes to this bug.