Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 1360402 - Clients can starve under heavy load
Summary: Clients can starve under heavy load
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: GlusterFS
Classification: Community
Component: io-threads
Version: mainline
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Jeff Darcy
QA Contact:
URL:
Whiteboard:
: 1360689 (view as bug list)
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-07-26 15:36 UTC by Jeff Darcy
Modified: 2017-03-09 20:34 UTC (History)
2 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2017-03-09 20:34:43 UTC
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:


Attachments (Terms of Use)

Description Jeff Darcy 2016-07-26 15:36:23 UTC
A user has reported an instance of $subject.

http://www.gluster.org/pipermail/gluster-users/2016-July/027355.html

We have long known this could be a problem, and the long-term solution is a proper QoS implementation.  In the interim, we should at least implement one of the measures described in a followup:

> We might be able to tweak io-threads (which already runs on the
> bricks and already has a global queue) to schedule requests in a
> fairer way across clients.  Right now it executes them in the
> same order that they were read from the network.  That tends to
> be a bit "unfair" and that should be fixed in the network code,
> but that's a much harder task.

Comment 1 Vijay Bellur 2016-07-26 15:37:57 UTC
REVIEW: http://review.gluster.org/14989 (io-threads: remove least-rate-limit option and code) posted (#4) for review on master by Jeff Darcy (jdarcy@redhat.com)

Comment 2 Vijay Bellur 2016-07-26 15:38:00 UTC
REVIEW: http://review.gluster.org/14904 (io-threads: distribute work fairly among clients) posted (#6) for review on master by Jeff Darcy (jdarcy@redhat.com)

Comment 3 Raghavendra G 2016-07-27 10:55:43 UTC
*** Bug 1360689 has been marked as a duplicate of this bug. ***

Comment 4 Vijay Bellur 2016-07-28 14:50:56 UTC
COMMIT: http://review.gluster.org/14989 committed in master by Jeff Darcy (jdarcy@redhat.com) 
------
commit 25d1bdc094a8408afc074817ec6d63c812c571f9
Author: Jeff Darcy <jdarcy@redhat.com>
Date:   Thu Jul 21 14:49:55 2016 -0400

    io-threads: remove least-rate-limit option and code
    
    This will be unnecessary, and mostly in the way, as real fairness
    guarantees are implemented.
    
    Change-Id: Ic61ec1c9e9add58385f1a4eafcfe2cc554ceefc8
    BUG: 1360402
    Signed-off-by: Jeff Darcy <jdarcy@redhat.com>
    Reviewed-on: http://review.gluster.org/14989
    Smoke: Gluster Build System <jenkins@build.gluster.org>
    Tested-by: Gluster Build System <jenkins@build.gluster.org>
    CentOS-regression: Gluster Build System <jenkins@build.gluster.org>
    NetBSD-regression: Gluster Build System <jenkins@build.gluster.org>

Comment 5 Vijay Bellur 2016-07-28 14:53:12 UTC
REVIEW: http://review.gluster.org/14904 (io-threads: distribute work fairly among clients) posted (#7) for review on master by Jeff Darcy (jdarcy@redhat.com)

Comment 6 Vijay Bellur 2016-07-28 18:52:10 UTC
COMMIT: http://review.gluster.org/14904 committed in master by Jeff Darcy (jdarcy@redhat.com) 
------
commit 1faf452818fe51fae1f07d17afab910a5e0da0cd
Author: Jeff Darcy <jdarcy@redhat.com>
Date:   Fri Jul 22 13:11:15 2016 -0400

    io-threads: distribute work fairly among clients
    
    This is the full "queue of queues" approach where each client gets its
    own queue (per priority) and we round-robin among them.
    
    Change-Id: I73955d1b9bb93f2ff781b48dfe2509009c519ec6
    BUG: 1360402
    Signed-off-by: Jeff Darcy <jdarcy@redhat.com>
    Reviewed-on: http://review.gluster.org/14904
    NetBSD-regression: NetBSD Build System <jenkins@build.gluster.org>
    CentOS-regression: Gluster Build System <jenkins@build.gluster.org>
    Smoke: Gluster Build System <jenkins@build.gluster.org>

Comment 7 Jeff Darcy 2017-03-09 20:34:43 UTC
This has been in for a while, long enough to be part of 3.10, so it's done.


Note You need to log in before you can comment on or make changes to this bug.