Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 162417 - (VM) Excessive swapping when free memory is ample
Summary: (VM) Excessive swapping when free memory is ample
Alias: None
Product: Red Hat Enterprise Linux 3
Classification: Red Hat
Component: kernel
Version: 3.0
Hardware: x86_64
OS: Linux
Target Milestone: ---
Assignee: Larry Woodman
QA Contact: Brian Brock
Depends On:
Blocks: 168424
TreeView+ depends on / blocked
Reported: 2005-07-04 11:44 UTC by Daniel Ramsay
Modified: 2007-11-30 22:07 UTC (History)
2 users (show)

Fixed In Version: RHSA-2006-0144
Doc Type: Bug Fix
Doc Text:
Clone Of:
Last Closed: 2006-03-15 16:11:53 UTC
Target Upstream Version:

Attachments (Terms of Use)
/proc/meminfo dump (deleted)
2005-07-04 11:45 UTC, Daniel Ramsay
no flags Details
output from "vmstat 1" (deleted)
2005-07-04 11:48 UTC, Daniel Ramsay
no flags Details

System ID Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2006:0144 qe-ready SHIPPED_LIVE Moderate: Updated kernel packages available for Red Hat Enterprise Linux 3 Update 7 2006-03-15 05:00:00 UTC

Description Daniel Ramsay 2005-07-04 11:44:35 UTC
From Bugzilla Helper:
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7.8) Gecko/20050511 Firefox/1.0.4

Description of problem:
Hi all, 

This report is very similar to:

The system runs for a short time and begins using up swap space in preference to free memory.  Swap space used rises sharply with system activity, when free memory stays fairly constant.

We're running Oracle (64-bit) on a cleanly installed Redhat AS 3.0 (64-bit update 5)

uname -a: 
Linux 2.4.21-32.ELsmp #1 SMP Fri Apr 15 21:03:28 EDT 2005 x86_64 x86_64 x86_64 GNU/Linux
The machine itself is a 4-way opteron with 16GB of ram.

Version-Release number of selected component (if applicable):

How reproducible:

Steps to Reproduce:
1. system boots
2. system starts oracle (64-bit) with SGA size 6.1 GB
3. system runs for an hour ...

[ sorry - quite straightforward ]

Actual Results:  Memory is swapped out after very little activity, despite an abundance of free memory.  Swap rate is proportional to system activity.  Without the skip_mapped_pages and pagecache tuning, the system would become unresponsive inside 2 hours.

Expected Results:  It should have used the free memory instead of swapping.  This was the case with the same database on the same machine running 32-bit Oracle & Linux.  It never touched the swapspace.

Additional info:

Attached is a vmstat 1 log for an affected time, plus meminfo dump.
This machine is our live database server.

We've stabilized the system with 

vm.skip_mapped_pages = 1
vm.pagecache = 1 10 10
vm.kswapd = 2048 64 64

gleaned from other bugzilla reports.

Currently it's running with a fluctuating amount of swap, hovering around 350MB, but can rise in seconds to 600MB.
It would be far preferable for it not to use the swap space at all.

Comment 1 Daniel Ramsay 2005-07-04 11:45:48 UTC
Created attachment 116326 [details]
/proc/meminfo dump

Comment 2 Daniel Ramsay 2005-07-04 11:48:31 UTC
Created attachment 116327 [details]
output from "vmstat 1"

Comment 3 Larry Woodman 2005-07-12 19:04:57 UTC
This is most likely a NUMA issue.  Please try adding "numa=off" at the end of
the boot command line in /boot/grub/grub.conf and verify that the problem goes
away with acceptable system performance.  Please let me know how this works out.

Thank-you, Larry Woodman

Comment 4 Daniel Ramsay 2005-07-13 14:29:02 UTC
This has worked out very well on our test system. Swap usage stays at zero even
after a greater workload has been thrown at it.  Thanks for your help!

Comment 5 Daniel Ramsay 2005-07-19 14:23:45 UTC
Sorry for the delay, I'm currently waiting for the DBA team to schedule a
switchover to the box that's been rebooted with the numa=off setting.  This
should happen in the next couple of days.

Comment 6 Daniel Ramsay 2005-08-16 08:24:18 UTC
Hi - 

We finally obtained a maintenance window for a server reboot.  The machine came
back perfectly, it's now using one very large memory region and no swap, even
under heavier load.  The throughput of the machine is better, as we'd expect
when it isn't swapping.   

Thanks very much for your help!

Comment 7 Ernie Petrides 2005-08-16 18:48:31 UTC
Thanks for the verification, Daniel.  Since you now have an easily
applied work-around, I'm going to close this as WONTFIX (since we've
decided not to change the default when this issue came up during U5).

This is essentially a duplicate of bug 130387.

Comment 8 Ernie Petrides 2005-11-30 07:48:15 UTC
A fix for this problem has just been committed to the RHEL3 U7
patch pool this evening (in kernel version 2.4.21-37.12.EL).

To enable an improved NUMA-friendly page allocation policy, please
set /proc/sys/vm/numa_memory_allocator via the "sysctl" command
(or put "vm.numa_memory_allocator = 1" in /etc/sysctl.conf).

Comment 12 Red Hat Bugzilla 2006-03-15 16:11:53 UTC
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on the solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

Note You need to log in before you can comment on or make changes to this bug.