Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 1065506 - nodejs supervisor consumes too much CPU
Summary: nodejs supervisor consumes too much CPU
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: OpenShift Online
Classification: Red Hat
Component: Image
Version: 2.x
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
: ---
Assignee: Michal Fojtik
QA Contact: libra bugs
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2014-02-14 19:48 UTC by Andy Grimm
Modified: 2016-11-08 03:47 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2014-03-12 03:07:30 UTC


Attachments (Terms of Use)

Description Andy Grimm 2014-02-14 19:48:01 UTC
Description of problem:

This is closely related to https://bugzilla.redhat.com/show_bug.cgi?id=1010461
and https://bugzilla.redhat.com/show_bug.cgi?id=1010464

In the first issue, I pointed out the recommendation that the default poll interval for nodejs supervisor be _at least_ 1000 ms.  That issue was resolved by changing the default to exactly 1000 ms, and we also made it tunable.  

However, we are still seeing that for large apps such as https://github.com/openshift-quickstart/openshift-ghost-quickstart , the supervisor process regularly takes 20% of a CPU (2/3 of a throttled user's quota).  That's just overhead that has nothing to do with serving the user's app, just watching the filesystem for changes.

I tested using "rhc set-env OPENSHIFT_NODEJS_POLL_INTERVAL=10000" on my app, an this reduced the CPU usage of supervisor to about 3%.  This is still too much in my opinion, but at least it leaves most of the cycles to the nodejs app. 

It would be preferable to set a much longer poll interval, since we know that the files should only ever change when a deploy happens.  We should easily be able to augment supervisor to only scan files in response to a signal

Version-Release number of selected component (if applicable):

openshift-origin-cartridge-nodejs-1.20.6-1.el6oso.noarch

How reproducible:

always

Steps to Reproduce:
1. rhc app create ghost nodejs-0.10 --env NODE_ENV=production --from-code https://github.com/openshift-quickstart/openshift-ghost-quickstart.git
2. rhc ssh ghost
3. watch "ps -ef" output

Actual results:

supervisor will take at least 15% of the CPU, more if other deps are added.
This is directly related to the number of files in app-root/runtime/repo/node_modules

Comment 1 Michal Fojtik 2014-02-17 09:12:37 UTC
Andy:  I'll send a PR to node-supervisor with a patch to make SIGUSR1 reload the files (shouldn't be difficult ;-)

In a meantime (supervisor version bump, PR accepted), I increased the POLL_INTERVAL to 10000.

Is it acceptable?

Comment 2 Michal Fojtik 2014-02-17 09:14:03 UTC
Bug 1065506 - Increase OPENSHIFT_NODEJS_POLL_INTERVAL default value to 10000

https://github.com/mfojtik/origin-server/commit/5ef076afb3ab186d9bb452ea87453d045475e584

Comment 3 Jianwei Hou 2014-02-17 11:16:08 UTC
Tested on devenv_4382(the commit was not merged, updated the code according to the commit manually), created one application with https://github.com/openshift-quickstart/openshift-ghost-quickstart.git template, the CPU usage is still now lower than 3%

[ghost-jhou.dev.rhcloud.com 5301ee42f2805d51e4000210]\> ps aux
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
1001      5854  0.7  0.6 1021216 23844 ?       Sl   06:11   0:01 node /opt/rh/nodejs010/root/usr/bin/supervisor -e node|js|coffee -p 10000
1001      5890  1.2  1.6 1043940 62560 ?       Sl   06:11   0:01 node index.js
1001      6933  0.0  0.0 104920  2680 ?        S    06:13   0:00 sshd: 5301ee42f2805d51e4000210@pts/1
1001      6934  0.4  0.0 108608  2044 pts/1    Ss   06:13   0:00 /bin/bash --init-file /usr/bin/rhcsh -i
1001      7519  0.0  0.0 110240  1140 pts/1    R+   06:14   0:00 ps aux

This bug could be verified, please move it to ON_QA, thanks

Comment 4 Michal Fojtik 2014-02-17 11:29:36 UTC
Thank you Hou Jianwei for testing this. Moving to ON_QA and merging the code.

Comment 5 Michal Fojtik 2014-02-17 12:28:15 UTC
There is also another related PR: https://github.com/openshift/origin-server/pull/4779

This change will allow you to set what files 'supervisor' should watch for changes. So in case you want to monitor just your application (not all dependencies/etc), you can set the application folder and server.js. This can help reduce the CPU load as well.

Comment 6 openshift-github-bot 2014-02-17 12:47:48 UTC
Commit pushed to master at https://github.com/openshift/origin-server

https://github.com/openshift/origin-server/commit/3b463a85f4165ccdb68d5a0fbc06fb1338299277
Bug 1065506 - Increase OPENSHIFT_NODEJS_POLL_INTERVAL default value to 10000

Comment 7 Andy Grimm 2014-02-17 13:57:17 UTC
Michal, thank you for the OpenShift changes.  I think those will be a huge help.  The supervisor PR for SIGUSR1 will be even better, but we can track that separately if you want.  Maybe we should do that via trello?

Comment 9 Meng Bo 2014-02-18 03:01:46 UTC
Checked on devenv-stage_700, the app created with the ghost quickstart will use about 1% CPU in total.

Move bug to verified.

[ghost-bmengdstage.dev.rhcloud.com 5302cb006c1ea611aa000007]\> ps ux
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
1000     11823  0.4  0.6 1021216 24512 ?       Sl   21:53   0:01 node /opt/rh/nodejs010/root/usr/bin/supervisor -e node|js|coffee -p 10000 -- index.js
1000     11858  0.5  1.7 1043940 63200 ?       Sl   21:53   0:01 node index.js
1000     14067  0.0  0.0 104920  2680 ?        S    21:58   0:00 sshd: 5302cb006c1ea611aa000007@pts/1
1000     14068  0.3  0.0 108572  2164 pts/1    Ss   21:58   0:00 /bin/bash --init-file /usr/bin/rhcsh -i
1000     14523  0.0  0.0 110232  1176 pts/1    R+   21:59   0:00 ps ux


Note You need to log in before you can comment on or make changes to this bug.