Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 1361681 - Memory leak with fuse
Summary: Memory leak with fuse
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: GlusterFS
Classification: Community
Component: fuse
Version: mainline
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: bugs@gluster.org
QA Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-07-29 17:57 UTC by Pranith Kumar K
Modified: 2016-07-30 09:51 UTC (History)
1 user (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2016-07-30 09:51:08 UTC
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:


Attachments (Terms of Use)

Description Pranith Kumar K 2016-07-29 17:57:29 UTC
Description of problem:
** time difference between 2 subsequent statedump of glusterfs(fuse mount) process is 2 hours **
statedump2:
[mount/fuse.fuse - usage-type gf_fuse_mt_iov_base memusage]
size=4294967290
num_allocs=2
max_size=4294967295
max_num_allocs=5
total_allocs=4007351


statedump 3
[mount/fuse.fuse - usage-type gf_fuse_mt_iov_base memusage]
size=4294967187
num_allocs=2
max_size=4294967295
max_num_allocs=5
total_allocs=8996888

statedump 4
[mount/fuse.fuse - usage-type gf_fuse_mt_iov_base memusage]
size=4294966952
num_allocs=1
max_size=4294967295
max_num_allocs=5
total_allocs=13664729

The mem_type gf_fuse_mt_iov_base consuming almost 4.29 GB but not exceeding the max size for gf_fuse_mt_iov_base. There was no significant increase in memory consumption by gf_fuse_mt_iov_base since statedump 2 but seems to be the reason for glusterfs process being picked up by OOM killer.

It also observed that the client system was showing 3.3GB free memory even though there was no rsync' was going on volume mount point. There may be possibility of high memory intensive processes running on the system as well.


In fuse_thread_proc() function:
                iov_in[0].iov_base = GF_CALLOC (1, msg0_size,
                                                gf_fuse_mt_iov_base);

                if (!iobuf || !iov_in[0].iov_base) {
                        gf_log (this->name, GF_LOG_ERROR,
                                "Out of memory");
                        if (iobuf)
                                iobuf_unref (iobuf);
                        GF_FREE (iov_in[0].iov_base);
                        sleep (10);
                        continue;
                }
......
                if (finh->opcode == FUSE_WRITE)
                        msg = iov_in[1].iov_base;
                else {
                        if (res > msg0_size) {
                                void *b = GF_REALLOC (iov_in[0].iov_base, res);
                                if (b) {
                                        iov_in[0].iov_base = b;
                                        finh = (fuse_in_header_t *)
                                                 iov_in[0].iov_base;
                                }
                                else {
                                        gf_log ("glusterfs-fuse", GF_LOG_ERROR,
                                                "Out of memory");
                                        send_fuse_err (this, finh, ENOMEM);

                                        goto cont_err;
                                }
                        }

It seems like we are leaking memory by over-riding iov_in[0].iov_base without freeing the existing memory-ptr in the variable?

Re-alloc may return different ptr than the one passed.

Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:

Comment 1 Vijay Bellur 2016-07-29 17:59:00 UTC
REVIEW: http://review.gluster.org/15052 (mount/fuse: Fix memory leak after re-alloc) posted (#1) for review on master by Pranith Kumar Karampuri (pkarampu@redhat.com)


Note You need to log in before you can comment on or make changes to this bug.