Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 1067284 - NFS smallfile performance comparison between 3.4.0.44rhs-1.el6rhs and 3.4.0.59rhs-1.el6rhs
Summary: NFS smallfile performance comparison between 3.4.0.44rhs-1.el6rhs and 3.4.0.5...
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: Red Hat Gluster Storage
Classification: Red Hat
Component: gluster-nfs
Version: 2.1
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
: ---
Assignee: Niels de Vos
QA Contact: storage-qa-internal@redhat.com
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2014-02-20 06:24 UTC by Anush Shetty
Modified: 2015-11-27 09:36 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2015-11-27 09:36:53 UTC


Attachments (Terms of Use)

Description Anush Shetty 2014-02-20 06:24:52 UTC
Description of problem: Below are the results from the NFS performance runs for 3.4.0.59rhs-1.el6rhs in comparison with U1 build - 3.4.0.44rhs-1.el6rhs 


Version-Release number of selected component (if applicable): 3.4.0.44rhs-1.el6rhs 


How reproducible: N/A


Steps to Reproduce:
1. Create a 2x2 Distributed Replicate volume and mount 2 NFS clients
2. Run smallfile workload https://github.com/bengland2/smallfile
3. Regression script used here - http://perf1.perf.lab.eng.bos.redhat.com/bengland/public/perf-data-collection/analysis/perf-reg-test/is-regression-v2.py

Actual results:

We see regression failures with some of the create, delete and metadata operations.


run106 - nfs - 3.4.0.44rhs-1.el6rhs - distrep - (quota no) - Threads:8, files:10000, file-size:64
run154 - nfs - 3.4.0.59rhs-1.el6rhs - distrep - (quota no) - Threads:8, files:10000, file-size:64

Operations                      RUN106  RUN154
-------------------------       ------- -------
create                          548     493
append                          1145    1038
read                            1661    1690
readdir                         9148    9256
ls-l                            6154    5823
mkdir                           813     746
rmdir                           632     558
stat                            4919    4335
chmod                           2233    2018
setxattr                        3429    2870
getxattr                        11456   11677
rename                          195     182
delete-renamed                  1446    1299


Regression script output:

======= Throughput create =========

decision parameters:
  sample type = throughput
  confidence threshold =  95.00 %
  max. pct. deviation =  10.00 %
  regression threshold =  10.00 % 
sample stats for baseline:
  min = 546.782261
  max = 552.384005
  mean = 549.382026
  sd = 2.822449
  pct.dev. =  0.51 %
sample stats for current:
  min = 493.648580
  max = 494.691803
  mean = 494.191710
  sd = 0.522941
  pct.dev. =  0.11 %
CHANGE -10.05 percent
magnitude of change is at least  9.53%
/usr/lib64/python2.6/site-packages/scipy/stats/stats.py:420: DeprecationWarning: scipy.stats.mean is deprecated; please update your code to use numpy.mean.
Please note that:
    - numpy.mean axis argument defaults to None, not 0
    - numpy.mean has a ddof argument to replace bias in a more general manner.
      scipy.stats.mean(a, bias=True) can be replaced by numpy.mean(x,
axis=0, ddof=1).
  axis=0, ddof=1).""", DeprecationWarning)
t-test t-statistic = 33.301841 probability = 0.000005
t-test says that mean of two sample sets differs with probability 100.00%
probability that sample sets have same mean = 0.0000
declaring a performance regression test FAILURE because of lower throughput
RESULT:10


======= Throughput append =========

decision parameters:
  sample type = throughput
  confidence threshold =  95.00 %
  max. pct. deviation =  10.00 %
  regression threshold =  10.00 % 
sample stats for baseline:
  min = 1131.253980
  max = 1156.436036
  mean = 1145.421079
  sd = 12.883555
  pct.dev. =  1.12 %
sample stats for current:
  min = 1029.065275
  max = 1049.452872
  mean = 1038.682042
  sd = 10.242677
  pct.dev. =  0.99 %
CHANGE -9.32 percent
magnitude of change is at least  8.19%
/usr/lib64/python2.6/site-packages/scipy/stats/stats.py:420: DeprecationWarning: scipy.stats.mean is deprecated; please update your code to use numpy.mean.
Please note that:
    - numpy.mean axis argument defaults to None, not 0
    - numpy.mean has a ddof argument to replace bias in a more general manner.
      scipy.stats.mean(a, bias=True) can be replaced by numpy.mean(x,
axis=0, ddof=1).
  axis=0, ddof=1).""", DeprecationWarning)
t-test t-statistic = 11.232610 probability = 0.000358
t-test says that mean of two sample sets differs with probability  99.96%
probability that sample sets have same mean = 0.0004
declaring a performance regression test FAILURE because of lower throughput
RESULT:10


======= Throughput read =========

decision parameters:
  sample type = throughput
  confidence threshold =  95.00 %
  max. pct. deviation =  10.00 %
  regression threshold =  10.00 % 
sample stats for baseline:
  min = 1625.178266
  max = 1707.806523
  mean = 1661.508432
  sd = 42.206361
  pct.dev. =  2.54 %
sample stats for current:
  min = 1652.487523
  max = 1711.558338
  mean = 1690.394255
  sd = 32.902546
  pct.dev. =  1.95 %
CHANGE  1.74 percent
magnitude of change is less than standard deviation of samples
/usr/lib64/python2.6/site-packages/scipy/stats/stats.py:420: DeprecationWarning: scipy.stats.mean is deprecated; please update your code to use numpy.mean.
Please note that:
    - numpy.mean axis argument defaults to None, not 0
    - numpy.mean has a ddof argument to replace bias in a more general manner.
      scipy.stats.mean(a, bias=True) can be replaced by numpy.mean(x,
axis=0, ddof=1).
  axis=0, ddof=1).""", DeprecationWarning)
t-test t-statistic = -0.934894 probability = 0.402770
t-test says that mean of two sample sets differs with probability  59.72%
probability that sample sets have same mean = 0.4028
sample sets are statistically indistinguishable for specified confidence level
RESULT:0


======= Throughput readdir =========

decision parameters:
  sample type = throughput
  confidence threshold =  95.00 %
  max. pct. deviation =  10.00 %
  regression threshold =  10.00 % 
sample stats for baseline:
  min = 8994.548775
  max = 9390.777368
  mean = 9148.915767
  sd = 212.110241
  pct.dev. =  2.32 %
sample stats for current:
  min = 9243.819577
  max = 9267.905076
  mean = 9257.024533
  sd = 12.209832
  pct.dev. =  0.13 %
CHANGE  1.18 percent
magnitude of change is less than standard deviation of samples
/usr/lib64/python2.6/site-packages/scipy/stats/stats.py:420: DeprecationWarning: scipy.stats.mean is deprecated; please update your code to use numpy.mean.
Please note that:
    - numpy.mean axis argument defaults to None, not 0
    - numpy.mean has a ddof argument to replace bias in a more general manner.
      scipy.stats.mean(a, bias=True) can be replaced by numpy.mean(x,
axis=0, ddof=1).
  axis=0, ddof=1).""", DeprecationWarning)
t-test t-statistic = -0.881336 probability = 0.427910
t-test says that mean of two sample sets differs with probability  57.21%
probability that sample sets have same mean = 0.4279
sample sets are statistically indistinguishable for specified confidence level
RESULT:0


======= Throughput ls-l =========

decision parameters:
  sample type = throughput
  confidence threshold =  95.00 %
  max. pct. deviation =  10.00 %
  regression threshold =  10.00 % 
sample stats for baseline:
  min = 3771.799332
  max = 7685.324100
  mean = 6155.155751
  sd = 2091.618139
  pct.dev. = 33.98 %
sample stats for current:
  min = 4518.678532
  max = 6671.560516
  mean = 5823.508967
  sd = 1146.826239
  pct.dev. = 19.69 %
ERROR: pct. deviation of 33.98 is too high for baseline samples
RESULT:11


======= Throughput mkdir =========

decision parameters:
  sample type = throughput
  confidence threshold =  95.00 %
  max. pct. deviation =  10.00 %
  regression threshold =  10.00 % 
sample stats for baseline:
  min = 749.377116
  max = 855.099996
  mean = 813.610125
  sd = 56.411608
  pct.dev. =  6.93 %
sample stats for current:
  min = 733.029315
  max = 762.575098
  mean = 746.807376
  sd = 14.873042
  pct.dev. =  1.99 %
CHANGE -8.21 percent
magnitude of change is at least  1.28%
/usr/lib64/python2.6/site-packages/scipy/stats/stats.py:420: DeprecationWarning: scipy.stats.mean is deprecated; please update your code to use numpy.mean.
Please note that:
    - numpy.mean axis argument defaults to None, not 0
    - numpy.mean has a ddof argument to replace bias in a more general manner.
      scipy.stats.mean(a, bias=True) can be replaced by numpy.mean(x,
axis=0, ddof=1).
  axis=0, ddof=1).""", DeprecationWarning)
t-test t-statistic = 1.983323 probability = 0.118351
t-test says that mean of two sample sets differs with probability  88.16%
probability that sample sets have same mean = 0.1184
sample sets are statistically indistinguishable for specified confidence level
RESULT:0


======= Throughput rmdir =========

decision parameters:
  sample type = throughput
  confidence threshold =  95.00 %
  max. pct. deviation =  10.00 %
  regression threshold =  10.00 % 
sample stats for baseline:
  min = 626.424265
  max = 641.345932
  mean = 632.532118
  sd = 7.820212
  pct.dev. =  1.24 %
sample stats for current:
  min = 554.047717
  max = 566.721321
  mean = 558.589879
  sd = 7.058134
  pct.dev. =  1.26 %
CHANGE -11.69 percent
magnitude of change is at least 10.43%
/usr/lib64/python2.6/site-packages/scipy/stats/stats.py:420: DeprecationWarning: scipy.stats.mean is deprecated; please update your code to use numpy.mean.
Please note that:
    - numpy.mean axis argument defaults to None, not 0
    - numpy.mean has a ddof argument to replace bias in a more general manner.
      scipy.stats.mean(a, bias=True) can be replaced by numpy.mean(x,
axis=0, ddof=1).
  axis=0, ddof=1).""", DeprecationWarning)
t-test t-statistic = 12.157510 probability = 0.000263
t-test says that mean of two sample sets differs with probability  99.97%
probability that sample sets have same mean = 0.0003
declaring a performance regression test FAILURE because of lower throughput
RESULT:10


======= Throughput stat =========

decision parameters:
  sample type = throughput
  confidence threshold =  95.00 %
  max. pct. deviation =  10.00 %
  regression threshold =  10.00 % 
sample stats for baseline:
  min = 4900.418965
  max = 4938.262781
  mean = 4919.315935
  sd = 18.921957
  pct.dev. =  0.38 %
sample stats for current:
  min = 4274.219088
  max = 4434.828514
  mean = 4335.970257
  sd = 86.495947
  pct.dev. =  1.99 %
CHANGE -11.86 percent
magnitude of change is at least  9.86%
/usr/lib64/python2.6/site-packages/scipy/stats/stats.py:420: DeprecationWarning: scipy.stats.mean is deprecated; please update your code to use numpy.mean.
Please note that:
    - numpy.mean axis argument defaults to None, not 0
    - numpy.mean has a ddof argument to replace bias in a more general manner.
      scipy.stats.mean(a, bias=True) can be replaced by numpy.mean(x,
axis=0, ddof=1).
  axis=0, ddof=1).""", DeprecationWarning)
t-test t-statistic = 11.411427 probability = 0.000336
t-test says that mean of two sample sets differs with probability  99.97%
probability that sample sets have same mean = 0.0003
declaring a performance regression test FAILURE because of lower throughput
RESULT:10


======= Throughput chmod =========

decision parameters:
  sample type = throughput
  confidence threshold =  95.00 %
  max. pct. deviation =  10.00 %
  regression threshold =  10.00 % 
sample stats for baseline:
  min = 2232.382665
  max = 2237.557171
  mean = 2234.263049
  sd = 2.862319
  pct.dev. =  0.13 %
sample stats for current:
  min = 1976.240010
  max = 2054.878358
  mean = 2019.279094
  sd = 39.843577
  pct.dev. =  1.97 %
CHANGE -9.62 percent
magnitude of change is at least  7.65%
/usr/lib64/python2.6/site-packages/scipy/stats/stats.py:420: DeprecationWarning: scipy.stats.mean is deprecated; please update your code to use numpy.mean.
Please note that:
    - numpy.mean axis argument defaults to None, not 0
    - numpy.mean has a ddof argument to replace bias in a more general manner.
      scipy.stats.mean(a, bias=True) can be replaced by numpy.mean(x,
axis=0, ddof=1).
  axis=0, ddof=1).""", DeprecationWarning)
t-test t-statistic = 9.321602 probability = 0.000737
t-test says that mean of two sample sets differs with probability  99.93%
probability that sample sets have same mean = 0.0007
declaring a performance regression test FAILURE because of lower throughput
RESULT:10


======= Throughput setxattr =========

decision parameters:
  sample type = throughput
  confidence threshold =  95.00 %
  max. pct. deviation =  10.00 %
  regression threshold =  10.00 % 
sample stats for baseline:
  min = 3400.581769
  max = 3457.681211
  mean = 3430.031261
  sd = 28.592225
  pct.dev. =  0.83 %
sample stats for current:
  min = 2807.907130
  max = 2933.084229
  mean = 2870.635283
  sd = 62.589017
  pct.dev. =  2.18 %
CHANGE -16.31 percent
magnitude of change is at least 14.13%
/usr/lib64/python2.6/site-packages/scipy/stats/stats.py:420: DeprecationWarning: scipy.stats.mean is deprecated; please update your code to use numpy.mean.
Please note that:
    - numpy.mean axis argument defaults to None, not 0
    - numpy.mean has a ddof argument to replace bias in a more general manner.
      scipy.stats.mean(a, bias=True) can be replaced by numpy.mean(x,
axis=0, ddof=1).
  axis=0, ddof=1).""", DeprecationWarning)
t-test t-statistic = 14.080710 probability = 0.000148
t-test says that mean of two sample sets differs with probability  99.99%
probability that sample sets have same mean = 0.0001
declaring a performance regression test FAILURE because of lower throughput
RESULT:10


======= Throughput getxattr =========

decision parameters:
  sample type = throughput
  confidence threshold =  95.00 %
  max. pct. deviation =  10.00 %
  regression threshold =  10.00 % 
sample stats for baseline:
  min = 11121.536186
  max = 11649.397065
  mean = 11457.270149
  sd = 291.764485
  pct.dev. =  2.55 %
sample stats for current:
  min = 11627.131972
  max = 11728.358597
  mean = 11677.918053
  sd = 50.614197
  pct.dev. =  0.43 %
CHANGE  1.93 percent
magnitude of change is less than standard deviation of samples
/usr/lib64/python2.6/site-packages/scipy/stats/stats.py:420: DeprecationWarning: scipy.stats.mean is deprecated; please update your code to use numpy.mean.
Please note that:
    - numpy.mean axis argument defaults to None, not 0
    - numpy.mean has a ddof argument to replace bias in a more general manner.
      scipy.stats.mean(a, bias=True) can be replaced by numpy.mean(x,
axis=0, ddof=1).
  axis=0, ddof=1).""", DeprecationWarning)
t-test t-statistic = -1.290594 probability = 0.266390
t-test says that mean of two sample sets differs with probability  73.36%
probability that sample sets have same mean = 0.2664
sample sets are statistically indistinguishable for specified confidence level
RESULT:0


======= Throughput rename =========

decision parameters:
  sample type = throughput
  confidence threshold =  95.00 %
  max. pct. deviation =  10.00 %
  regression threshold =  10.00 % 
sample stats for baseline:
  min = 195.527511
  max = 195.659466
  mean = 195.605364
  sd = 0.069109
  pct.dev. =  0.04 %
sample stats for current:
  min = 182.264552
  max = 183.033306
  mean = 182.527871
  sd = 0.437848
  pct.dev. =  0.24 %
CHANGE -6.69 percent
magnitude of change is at least  6.45%
/usr/lib64/python2.6/site-packages/scipy/stats/stats.py:420: DeprecationWarning: scipy.stats.mean is deprecated; please update your code to use numpy.mean.
Please note that:
    - numpy.mean axis argument defaults to None, not 0
    - numpy.mean has a ddof argument to replace bias in a more general manner.
      scipy.stats.mean(a, bias=True) can be replaced by numpy.mean(x,
axis=0, ddof=1).
  axis=0, ddof=1).""", DeprecationWarning)
t-test t-statistic = 51.099726 probability = 0.000001
t-test says that mean of two sample sets differs with probability 100.00%
probability that sample sets have same mean = 0.0000
RESULT:0


======= Throughput delete-renamed =========

decision parameters:
  sample type = throughput
  confidence threshold =  95.00 %
  max. pct. deviation =  10.00 %
  regression threshold =  10.00 % 
sample stats for baseline:
  min = 1440.128285
  max = 1455.358596
  mean = 1446.351744
  sd = 7.987556
  pct.dev. =  0.55 %
sample stats for current:
  min = 1284.636443
  max = 1314.856695
  mean = 1299.809573
  sd = 15.110520
  pct.dev. =  1.16 %
CHANGE -10.13 percent
magnitude of change is at least  8.97%
/usr/lib64/python2.6/site-packages/scipy/stats/stats.py:420: DeprecationWarning: scipy.stats.mean is deprecated; please update your code to use numpy.mean.
Please note that:
    - numpy.mean axis argument defaults to None, not 0
    - numpy.mean has a ddof argument to replace bias in a more general manner.
      scipy.stats.mean(a, bias=True) can be replaced by numpy.mean(x,
axis=0, ddof=1).
  axis=0, ddof=1).""", DeprecationWarning)
t-test t-statistic = 14.850327 probability = 0.000120
t-test says that mean of two sample sets differs with probability  99.99%
probability that sample sets have same mean = 0.0001
declaring a performance regression test FAILURE because of lower throughput





Expected results:


Additional info:

 
Volume Name: testvol
Type: Distributed-Replicate
Volume ID: 08c3f51a-0526-4388-b7fd-a8714e0e8bdb
Status: Started
Number of Bricks: 2 x 2 = 4
Transport-type: tcp
Bricks:
Brick1: rhs-gp-srv2.lab.eng.blr.redhat.com:/data/datastore
Brick2: rhs-gp-srv5.lab.eng.blr.redhat.com:/data/datastore
Brick3: rhs-gp-srv6.lab.eng.blr.redhat.com:/data/datastore
Brick4: rhs-gp-srv16.lab.eng.blr.redhat.com:/data/datastore
Options Reconfigured:
nfs.mem-factor: 20
performance.quick-read: on
performance.open-behind: on

Comment 2 santosh pradhan 2014-06-18 06:52:24 UTC
Same update as BZ 1059989.

Because of I/O throttling fix (BZ 977492) which was gone into 3.4.0.58rhs-1.el6, the performance might be impacted. Your workload was not very big (-s 10g -t 8 which means 8 threads write of 10g size I/O = 80g). (See BZ 977492, because very big I/O load, NFS hangs as it was not able to handle the I/O load).

I/O throttling can be turned OFF by:

gluster volume set <volname> nfs.outstanding-rpc-limit 0
gluster volume set <volname> server.outstanding-rpc-limit 0

And restart the I/O workload, ideally you should get the same performance.

It should not be a bug.

Thanks,
Santosh

Comment 3 santosh pradhan 2014-07-22 17:39:32 UTC
There is no update in a month. Should it be closed?

Comment 4 Anush Shetty 2014-07-23 03:37:19 UTC
Running the tests with the tunables. Will update soon.


Note You need to log in before you can comment on or make changes to this bug.