Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 1067289 - FUSE smallfile performance comparison between 3.4.0.44rhs-1.el6rhs and 3.4.0.59rhs-1.el6rhs
Summary: FUSE smallfile performance comparison between 3.4.0.44rhs-1.el6rhs and 3.4.0....
Keywords:
Status: CLOSED EOL
Alias: None
Product: Red Hat Gluster Storage
Classification: Red Hat
Component: glusterfs
Version: 2.1
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
: ---
Assignee: Nagaprasad Sathyanarayana
QA Contact: storage-qa-internal@redhat.com
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2014-02-20 06:44 UTC by Anush Shetty
Modified: 2016-02-18 00:20 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2015-12-03 17:18:26 UTC


Attachments (Terms of Use)

Description Anush Shetty 2014-02-20 06:44:46 UTC
Description of problem: Below are the results from the FUSE performance runs for 3.4.0.59rhs-1.el6rhs in comparison with U1 build - 3.4.0.44rhs-1.el6rhs 


Version-Release number of selected component (if applicable): 3.4.0.44rhs-1.el6rhs 


Steps to Reproduce:
1. Create a 2x2 Distributed Replicate volume and mount 2 NFS clients
2. Run smallfile workload https://github.com/bengland2/smallfile
3. Regression script used here - http://perf1.perf.lab.eng.bos.redhat.com/bengland/public/perf-data-collection/analysis/perf-reg-test/is-regression-v2.py

Actual results:

See regression with mkdir, metadata and delete operations.

# ./calc_avg 107 153

run107 - glusterfs - 3.4.0.44rhs-1.el6rhs - distrep - (quota no) - Threads:8, files:10000, file-size:64
run153 - glusterfs - 3.4.0.59rhs-1.el6rhs - distrep - (quota no) - Threads:8, files:10000, file-size:64

Operations                      RUN107  RUN153
-------------------------       ------- -------
create                          949     1304
append                          1155    1643
read                            3145    3125
readdir                         9344    9292
ls-l                            9023    9148
mkdir                           1558    1171
rmdir                           1213    939
stat                            6862    6619
chmod                           3284    2721
setxattr                        5496    5492
getxattr                        10958   11063
rename                          394     351
delete-renamed                  2945    2544

Regression script output:

======= Throughput create =========

decision parameters:
  sample type = throughput
  confidence threshold =  95.00 %
  max. pct. deviation =  10.00 %
  regression threshold =  10.00 % 
sample stats for baseline:
  min = 935.206374
  max = 965.272422
  mean = 949.974477
  sd = 15.040025
  pct.dev. =  1.58 %
sample stats for current:
  min = 1303.113894
  max = 1306.837339
  mean = 1304.745412
  sd = 1.903942
  pct.dev. =  0.15 %
CHANGE 37.35 percent
magnitude of change is at least 35.76%
/usr/lib64/python2.6/site-packages/scipy/stats/stats.py:420: DeprecationWarning: scipy.stats.mean is deprecated; please update your code to use numpy.mean.
Please note that:
    - numpy.mean axis argument defaults to None, not 0
    - numpy.mean has a ddof argument to replace bias in a more general manner.
      scipy.stats.mean(a, bias=True) can be replaced by numpy.mean(x,
axis=0, ddof=1).
  axis=0, ddof=1).""", DeprecationWarning)
t-test t-statistic = -40.532912 probability = 0.000002
t-test says that mean of two sample sets differs with probability 100.00%
probability that sample sets have same mean = 0.0000
current sample set is statistically better than baseline sample set
RESULT:0


======= Throughput append =========

decision parameters:
  sample type = throughput
  confidence threshold =  95.00 %
  max. pct. deviation =  10.00 %
  regression threshold =  10.00 % 
sample stats for baseline:
  min = 1140.964188
  max = 1164.293130
  mean = 1155.759447
  sd = 12.863335
  pct.dev. =  1.11 %
sample stats for current:
  min = 1615.236761
  max = 1660.977720
  mean = 1643.901768
  sd = 24.975758
  pct.dev. =  1.52 %
CHANGE 42.24 percent
magnitude of change is at least 40.72%
/usr/lib64/python2.6/site-packages/scipy/stats/stats.py:420: DeprecationWarning: scipy.stats.mean is deprecated; please update your code to use numpy.mean.
Please note that:
    - numpy.mean axis argument defaults to None, not 0
    - numpy.mean has a ddof argument to replace bias in a more general manner.
      scipy.stats.mean(a, bias=True) can be replaced by numpy.mean(x,
axis=0, ddof=1).
  axis=0, ddof=1).""", DeprecationWarning)
t-test t-statistic = -30.095304 probability = 0.000007
t-test says that mean of two sample sets differs with probability 100.00%
probability that sample sets have same mean = 0.0000
current sample set is statistically better than baseline sample set
RESULT:0


======= Throughput read =========

decision parameters:
  sample type = throughput
  confidence threshold =  95.00 %
  max. pct. deviation =  10.00 %
  regression threshold =  10.00 % 
sample stats for baseline:
  min = 3127.617983
  max = 3154.921747
  mean = 3145.769274
  sd = 15.719667
  pct.dev. =  0.50 %
sample stats for current:
  min = 3121.720421
  max = 3132.022800
  mean = 3125.665536
  sd = 5.558651
  pct.dev. =  0.18 %
CHANGE -0.64 percent
magnitude of change is at least  0.14%
/usr/lib64/python2.6/site-packages/scipy/stats/stats.py:420: DeprecationWarning: scipy.stats.mean is deprecated; please update your code to use numpy.mean.
Please note that:
    - numpy.mean axis argument defaults to None, not 0
    - numpy.mean has a ddof argument to replace bias in a more general manner.
      scipy.stats.mean(a, bias=True) can be replaced by numpy.mean(x,
axis=0, ddof=1).
  axis=0, ddof=1).""", DeprecationWarning)
t-test t-statistic = 2.088382 probability = 0.105023
t-test says that mean of two sample sets differs with probability  89.50%
probability that sample sets have same mean = 0.1050
sample sets are statistically indistinguishable for specified confidence level
RESULT:0


======= Throughput readdir =========

decision parameters:
  sample type = throughput
  confidence threshold =  95.00 %
  max. pct. deviation =  10.00 %
  regression threshold =  10.00 % 
sample stats for baseline:
  min = 9211.949712
  max = 9459.987195
  mean = 9344.677937
  sd = 124.932836
  pct.dev. =  1.34 %
sample stats for current:
  min = 9264.187219
  max = 9321.923684
  mean = 9292.395374
  sd = 28.890863
  pct.dev. =  0.31 %
CHANGE -0.56 percent
magnitude of change is less than standard deviation of samples
/usr/lib64/python2.6/site-packages/scipy/stats/stats.py:420: DeprecationWarning: scipy.stats.mean is deprecated; please update your code to use numpy.mean.
Please note that:
    - numpy.mean axis argument defaults to None, not 0
    - numpy.mean has a ddof argument to replace bias in a more general manner.
      scipy.stats.mean(a, bias=True) can be replaced by numpy.mean(x,
axis=0, ddof=1).
  axis=0, ddof=1).""", DeprecationWarning)
t-test t-statistic = 0.706201 probability = 0.519025
t-test says that mean of two sample sets differs with probability  48.10%
probability that sample sets have same mean = 0.5190
sample sets are statistically indistinguishable for specified confidence level
RESULT:0


======= Throughput ls-l =========

decision parameters:
  sample type = throughput
  confidence threshold =  95.00 %
  max. pct. deviation =  10.00 %
  regression threshold =  10.00 % 
sample stats for baseline:
  min = 8908.896360
  max = 9222.100015
  mean = 9023.451334
  sd = 172.707690
  pct.dev. =  1.91 %
sample stats for current:
  min = 9127.001799
  max = 9176.475208
  mean = 9148.586393
  sd = 25.332034
  pct.dev. =  0.28 %
CHANGE  1.39 percent
magnitude of change is less than standard deviation of samples
/usr/lib64/python2.6/site-packages/scipy/stats/stats.py:420: DeprecationWarning: scipy.stats.mean is deprecated; please update your code to use numpy.mean.
Please note that:
    - numpy.mean axis argument defaults to None, not 0
    - numpy.mean has a ddof argument to replace bias in a more general manner.
      scipy.stats.mean(a, bias=True) can be replaced by numpy.mean(x,
axis=0, ddof=1).
  axis=0, ddof=1).""", DeprecationWarning)
t-test t-statistic = -1.241669 probability = 0.282192
t-test says that mean of two sample sets differs with probability  71.78%
probability that sample sets have same mean = 0.2822
sample sets are statistically indistinguishable for specified confidence level
RESULT:0


======= Throughput mkdir =========

decision parameters:
  sample type = throughput
  confidence threshold =  95.00 %
  max. pct. deviation =  10.00 %
  regression threshold =  10.00 % 
sample stats for baseline:
  min = 1543.281113
  max = 1583.676238
  mean = 1558.479120
  sd = 21.975627
  pct.dev. =  1.41 %
sample stats for current:
  min = 1163.288837
  max = 1181.111898
  mean = 1172.086392
  sd = 8.913717
  pct.dev. =  0.76 %
CHANGE -24.79 percent
magnitude of change is at least 23.38%
/usr/lib64/python2.6/site-packages/scipy/stats/stats.py:420: DeprecationWarning: scipy.stats.mean is deprecated; please update your code to use numpy.mean.
Please note that:
    - numpy.mean axis argument defaults to None, not 0
    - numpy.mean has a ddof argument to replace bias in a more general manner.
      scipy.stats.mean(a, bias=True) can be replaced by numpy.mean(x,
axis=0, ddof=1).
  axis=0, ddof=1).""", DeprecationWarning)
t-test t-statistic = 28.221082 probability = 0.000009
t-test says that mean of two sample sets differs with probability 100.00%
probability that sample sets have same mean = 0.0000
declaring a performance regression test FAILURE because of lower throughput
RESULT:10


======= Throughput rmdir =========

decision parameters:
  sample type = throughput
  confidence threshold =  95.00 %
  max. pct. deviation =  10.00 %
  regression threshold =  10.00 % 
sample stats for baseline:
  min = 1205.394687
  max = 1228.121770
  mean = 1213.802388
  sd = 12.463589
  pct.dev. =  1.03 %
sample stats for current:
  min = 927.824854
  max = 951.477068
  mean = 940.001276
  sd = 11.841662
  pct.dev. =  1.26 %
CHANGE -22.56 percent
magnitude of change is at least 21.30%
/usr/lib64/python2.6/site-packages/scipy/stats/stats.py:420: DeprecationWarning: scipy.stats.mean is deprecated; please update your code to use numpy.mean.
Please note that:
    - numpy.mean axis argument defaults to None, not 0
    - numpy.mean has a ddof argument to replace bias in a more general manner.
      scipy.stats.mean(a, bias=True) can be replaced by numpy.mean(x,
axis=0, ddof=1).
  axis=0, ddof=1).""", DeprecationWarning)
t-test t-statistic = 27.584720 probability = 0.000010
t-test says that mean of two sample sets differs with probability 100.00%
probability that sample sets have same mean = 0.0000
declaring a performance regression test FAILURE because of lower throughput
RESULT:10


======= Throughput stat =========

decision parameters:
  sample type = throughput
  confidence threshold =  95.00 %
  max. pct. deviation =  10.00 %
  regression threshold =  10.00 % 
sample stats for baseline:
  min = 6826.844657
  max = 6888.968226
  mean = 6863.091108
  sd = 32.333833
  pct.dev. =  0.47 %
sample stats for current:
  min = 6588.337912
  max = 6649.200892
  mean = 6619.541972
  sd = 30.460896
  pct.dev. =  0.46 %
CHANGE -3.55 percent
magnitude of change is at least  3.08%
/usr/lib64/python2.6/site-packages/scipy/stats/stats.py:420: DeprecationWarning: scipy.stats.mean is deprecated; please update your code to use numpy.mean.
Please note that:
    - numpy.mean axis argument defaults to None, not 0
    - numpy.mean has a ddof argument to replace bias in a more general manner.
      scipy.stats.mean(a, bias=True) can be replaced by numpy.mean(x,
axis=0, ddof=1).
  axis=0, ddof=1).""", DeprecationWarning)
t-test t-statistic = 9.496114 probability = 0.000686
t-test says that mean of two sample sets differs with probability  99.93%
probability that sample sets have same mean = 0.0007
RESULT:0


======= Throughput chmod =========

decision parameters:
  sample type = throughput
  confidence threshold =  95.00 %
  max. pct. deviation =  10.00 %
  regression threshold =  10.00 % 
sample stats for baseline:
  min = 3172.716293
  max = 3379.210307
  mean = 3284.863788
  sd = 104.391573
  pct.dev. =  3.18 %
sample stats for current:
  min = 2711.253229
  max = 2732.296283
  mean = 2721.358353
  sd = 10.546218
  pct.dev. =  0.39 %
CHANGE -17.15 percent
magnitude of change is at least 13.98%
/usr/lib64/python2.6/site-packages/scipy/stats/stats.py:420: DeprecationWarning: scipy.stats.mean is deprecated; please update your code to use numpy.mean.
Please note that:
    - numpy.mean axis argument defaults to None, not 0
    - numpy.mean has a ddof argument to replace bias in a more general manner.
      scipy.stats.mean(a, bias=True) can be replaced by numpy.mean(x,
axis=0, ddof=1).
  axis=0, ddof=1).""", DeprecationWarning)
t-test t-statistic = 9.302256 probability = 0.000743
t-test says that mean of two sample sets differs with probability  99.93%
probability that sample sets have same mean = 0.0007
declaring a performance regression test FAILURE because of lower throughput
RESULT:10


======= Throughput setxattr =========

decision parameters:
  sample type = throughput
  confidence threshold =  95.00 %
  max. pct. deviation =  10.00 %
  regression threshold =  10.00 % 
sample stats for baseline:
  min = 5420.509941
  max = 5539.888072
  mean = 5496.768085
  sd = 66.229809
  pct.dev. =  1.20 %
sample stats for current:
  min = 5478.421349
  max = 5514.647063
  mean = 5492.831305
  sd = 19.214839
  pct.dev. =  0.35 %
CHANGE -0.07 percent
magnitude of change is less than standard deviation of samples
/usr/lib64/python2.6/site-packages/scipy/stats/stats.py:420: DeprecationWarning: scipy.stats.mean is deprecated; please update your code to use numpy.mean.
Please note that:
    - numpy.mean axis argument defaults to None, not 0
    - numpy.mean has a ddof argument to replace bias in a more general manner.
      scipy.stats.mean(a, bias=True) can be replaced by numpy.mean(x,
axis=0, ddof=1).
  axis=0, ddof=1).""", DeprecationWarning)
t-test t-statistic = 0.098878 probability = 0.925992
t-test says that mean of two sample sets differs with probability   7.40%
probability that sample sets have same mean = 0.9260
sample sets are statistically indistinguishable for specified confidence level
RESULT:0


======= Throughput getxattr =========

decision parameters:
  sample type = throughput
  confidence threshold =  95.00 %
  max. pct. deviation =  10.00 %
  regression threshold =  10.00 % 
sample stats for baseline:
  min = 10912.316286
  max = 11014.741587
  mean = 10959.318809
  sd = 51.729209
  pct.dev. =  0.47 %
sample stats for current:
  min = 10955.457858
  max = 11130.276796
  mean = 11063.928061
  sd = 94.715779
  pct.dev. =  0.86 %
CHANGE  0.95 percent
magnitude of change is at least  0.10%
/usr/lib64/python2.6/site-packages/scipy/stats/stats.py:420: DeprecationWarning: scipy.stats.mean is deprecated; please update your code to use numpy.mean.
Please note that:
    - numpy.mean axis argument defaults to None, not 0
    - numpy.mean has a ddof argument to replace bias in a more general manner.
      scipy.stats.mean(a, bias=True) can be replaced by numpy.mean(x,
axis=0, ddof=1).
  axis=0, ddof=1).""", DeprecationWarning)
t-test t-statistic = -1.678896 probability = 0.168473
t-test says that mean of two sample sets differs with probability  83.15%
probability that sample sets have same mean = 0.1685
sample sets are statistically indistinguishable for specified confidence level
RESULT:0


======= Throughput rename =========

decision parameters:
  sample type = throughput
  confidence threshold =  95.00 %
  max. pct. deviation =  10.00 %
  regression threshold =  10.00 % 
sample stats for baseline:
  min = 394.679407
  max = 396.027252
  mean = 395.143323
  sd = 0.765820
  pct.dev. =  0.19 %
sample stats for current:
  min = 350.602284
  max = 353.179152
  mean = 351.494232
  sd = 1.460023
  pct.dev. =  0.42 %
CHANGE -11.05 percent
magnitude of change is at least 10.63%
/usr/lib64/python2.6/site-packages/scipy/stats/stats.py:420: DeprecationWarning: scipy.stats.mean is deprecated; please update your code to use numpy.mean.
Please note that:
    - numpy.mean axis argument defaults to None, not 0
    - numpy.mean has a ddof argument to replace bias in a more general manner.
      scipy.stats.mean(a, bias=True) can be replaced by numpy.mean(x,
axis=0, ddof=1).
  axis=0, ddof=1).""", DeprecationWarning)
t-test t-statistic = 45.856341 probability = 0.000001
t-test says that mean of two sample sets differs with probability 100.00%
probability that sample sets have same mean = 0.0000
declaring a performance regression test FAILURE because of lower throughput
RESULT:10


======= Throughput delete-renamed =========

decision parameters:
  sample type = throughput
  confidence threshold =  95.00 %
  max. pct. deviation =  10.00 %
  regression threshold =  10.00 % 
sample stats for baseline:
  min = 2785.621090
  max = 3085.223262
  mean = 2946.163113
  sd = 150.951875
  pct.dev. =  5.12 %
sample stats for current:
  min = 2481.205431
  max = 2612.190555
  mean = 2544.669401
  sd = 65.586746
  pct.dev. =  2.58 %
CHANGE -13.63 percent
magnitude of change is at least  8.50%
/usr/lib64/python2.6/site-packages/scipy/stats/stats.py:420: DeprecationWarning: scipy.stats.mean is deprecated; please update your code to use numpy.mean.
Please note that:
    - numpy.mean axis argument defaults to None, not 0
    - numpy.mean has a ddof argument to replace bias in a more general manner.
      scipy.stats.mean(a, bias=True) can be replaced by numpy.mean(x,
axis=0, ddof=1).
  axis=0, ddof=1).""", DeprecationWarning)
t-test t-statistic = 4.225228 probability = 0.013421
t-test says that mean of two sample sets differs with probability  98.66%
probability that sample sets have same mean = 0.0134
declaring a performance regression test FAILURE because of lower throughput

Expected results:


Additional info:

Volume Name: testvol
Type: Distributed-Replicate
Volume ID: 34063471-5225-460d-81d1-113636274a36
Status: Started
Number of Bricks: 2 x 2 = 4
Transport-type: tcp
Bricks:
Brick1: rhs-gp-srv2.lab.eng.blr.redhat.com:/data/datastore
Brick2: rhs-gp-srv5.lab.eng.blr.redhat.com:/data/datastore
Brick3: rhs-gp-srv6.lab.eng.blr.redhat.com:/data/datastore
Brick4: rhs-gp-srv16.lab.eng.blr.redhat.com:/data/datastore
Options Reconfigured:
performance.quick-read: on
performance.open-behind: on

Comment 3 Vivek Agarwal 2015-12-03 17:18:26 UTC
Thank you for submitting this issue for consideration in Red Hat Gluster Storage. The release for which you requested us to review, is now End of Life. Please See https://access.redhat.com/support/policy/updates/rhs/

If you can reproduce this bug against a currently maintained version of Red Hat Gluster Storage, please feel free to file a new report against the current release.


Note You need to log in before you can comment on or make changes to this bug.