Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 1060871 - Tuning Recommendation for Apache to use Worker
Summary: Tuning Recommendation for Apache to use Worker
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Documentation
Version: 2.2.0
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
: ---
Assignee: Julie
QA Contact: brice
URL:
Whiteboard:
Depends On: 1061411
Blocks:
TreeView+ depends on / blocked
 
Reported: 2014-02-03 19:26 UTC by Eric Rich
Modified: 2018-12-04 17:18 UTC (History)
8 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2014-08-07 04:10:32 UTC
Target Upstream Version:
juwu: needinfo+


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Red Hat Knowledge Base (Solution) 25930 None None None Never
Red Hat Knowledge Base (Solution) 543133 None None None Never

Description Eric Rich 2014-02-03 19:26:59 UTC
Description of problem:

We should let customer know that using Apache's worker is preferred over using prefork.

Comment 4 Luke Meyer 2014-02-04 19:34:25 UTC
Currently blocked by the need for supported idler under worker MPM.

Worker MPM has the benefit of sharing the config definition (which can be substantial) between all threads in a process, reducing RAM footprint. There may be some benefits in reducing spin-up/spin-down churn under varying load, as well as proxy connection pooling, but I can't attest to that. OpenShift Online uses this MPM.

Comment 5 Eric Rich 2014-03-17 16:32:34 UTC
https://access.redhat.com/labs/lbconfig/ can probably help with this, all in all I think it needs to be modified a bit to explain openshift, and we may want to check with Operations on there configuration and why the settings are the way they are. 

However https://access.redhat.com/labs/lbconfig/#/?apache_cores=4&apache_instances_per_server=1&apache_servers=1&jboss_cores=1&jboss_jvms=1&jboss_servers=1&long_running=false&mpmtype=Worker&modtype=mod_proxy&jboss_version=6&firewall=false&same_server=true I feel gets us close, and is something we can suggest to customers (in a consistent manner with our other tooling on the subject). 

The key here is setting 'Cores Per Server' in the Apache Configuration to the number of CPU cores for your node. All other setting should be 1 with long running request being set to some number (as defined by the customer).

Comment 6 Julie 2014-06-27 05:35:17 UTC
Hi Luke,
   Can you provide me a bit more on info what needs to be documented here?

Many thanks,
Julie

Comment 7 Luke Meyer 2014-06-30 14:57:33 UTC
Julie, thanks for bumping this. We could actually proceed with this now as of OSE 2.1.

At issue is the MPM used for the OpenShift node front-end HTTP proxy, i.e. the component through which standard requests to applications pass (leaving off websockets and SNI proxy requests). This is currently httpd (Apache) and the default MPM is the single-threaded "prefork" module. The recommendation is to change this default to use the multi-threaded "worker" module. Changing is fairly simple (https://access.redhat.com/site/solutions/25930). I feel like this recommendation is a good idea, but I'm not sure we've quantified why.

For some customers, use of worker is more or less mandated by their need to support third-party httpd modules which require it. Prior to OSE 2.1 this wasn't supportable in conjunction with the node idler as the idler was based on PHP and Red Hat doesn't support a multi-threaded mod_php. With OSE 2.1 this was rewritten in Perl so either MPM should be readily supported.

Aside from compatibility requirements, I think the worker module is better-suited than prefork for the following reasons:
1. Multiple threads (per worker process) share the memory used for configuration rather than replicating it per-process (as in prefork). Given our config can grow with hundreds of gears, this can be a substantial savings.
2. There is less process creation and destruction when workers are allocated in blocks of threads as opposed to per-process. Given the workers usually do nothing themselves except proxy requests, the worker MPM should free up more CPU for the actual backend.
3. With multiple threads it is possible to implement useful connection pooling (when using the apache-vhost plugin, which is also not default at this time), enabling KeepAlive and reuse of connections all the way to the gear application (currently every connection is torn down after use). In most environments this should reduce latency and CPU use on proxy connections to the backend, as well as TIME_WAIT connection exhaustion.

The problem is that these benefits haven't been clearly quantified in OSE performance tests, and are heavily dependent on usage and tuning. I don't think the app Eric linked (https://access.redhat.com/labs/lbconfig/) is actually very relevant as it's for a specific load balancing situation, and instead we have a proxy to a large number of backend apps. Ideally we would have something like that specifically for OpenShift which could help make tuning recommendations for the httpd MPM and OSE frontend plugin.

So at this time we can clearly indicate that the worker MPM is supported as of OSE 2.1. And I would consider it a recommended, but we haven't rigorously proved the benefits or indicated how to tune to take advantage of it. Eric, Jeremy, and data, even anecdotal?

Comment 9 Luke Meyer 2014-06-30 15:07:12 UTC
s/and data/any data/

Comment 12 Julie 2014-07-03 00:02:31 UTC
Eric,
  Pls also let me know if you have any feedback. My understanding is the worker module is probably the recommended one, but it really depends on the use case. I've added a note saying both modules are supported, so the admin can decide which one to use.

Kind regards,
Julie

Comment 13 Luke Meyer 2014-07-03 04:37:22 UTC
"... both the process-based prefork model and the thread-based worker model in Apache are supported."

First, s/model/module/ everywhere. These are alternative Multi-Processing Modules.

Also it may be a little pedantic but the more accurate distinction is single- versus multi-threaded:

"... both the single-threaded prefork module and the multi-threaded worker module in Apache httpd are supported."

Otherwise... it falls short of a recommendation, but at least it raises awareness of the option. Maybe it could be a little stronger:

"The prefork module is used by default, but for many OpenShift use cases, the worker module may perform better. "

Perhaps we should spawn a story to do performance testing and feed that into tuning recommendations and cloning "lbconfig" to make automated recommendations.

Comment 14 Eric Rich 2014-07-03 13:45:14 UTC
(In reply to Julie from comment #12)
> Eric,
>   Pls also let me know if you have any feedback. My understanding is the
> worker module is probably the recommended one, but it really depends on the
> use case. I've added a note saying both modules are supported, so the admin
> can decide which one to use.
> 
> Kind regards,
> Julie

It looks good following Luke's comments

Comment 17 Julie 2014-07-07 01:26:45 UTC
typos fixed.
Thanks brice for your review.

Cheers,
Julie


Note You need to log in before you can comment on or make changes to this bug.