Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 1513692 - io-stats appends now instead of overwriting which floods filesystem with logs
Summary: io-stats appends now instead of overwriting which floods filesystem with logs
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: GlusterFS
Classification: Community
Component: io-stats
Version: mainline
Hardware: x86_64
OS: Linux
unspecified
urgent
Target Milestone: ---
Assignee: bugs@gluster.org
QA Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-11-15 19:11 UTC by Chris Holcombe
Modified: 2018-03-15 11:20 UTC (History)
3 users (show)

Fixed In Version: glusterfs-4.0.0
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-03-15 11:20:54 UTC
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:


Attachments (Terms of Use)

Description Chris Holcombe 2017-11-15 19:11:02 UTC
Description of problem:
The io-stats xlator used to have a behavior where it constantly overwrites the files every x seconds.  This behavior has changed in 3.12.x to now append to the file.  After enabling the xlator the filesystem on my cluster filled up almost immediately and crashed.  

Version-Release number of selected component (if applicable):
3.12.0 and newer

How reproducible:
Always.  

Steps to Reproduce:
1. gluster volume set gv0 diagnostics.latency-measurement on
2. gluster volume set gv0 diagnostics.count-fop-hits on
3. gluster volume set gv0 diagnostics.stats-dump-interval 10
4. gluster volume set gv0 diagnostics.fop-sample-interval 5

Actual results:
Stats files in /var/lib/glusterd/stats fill up at an alarming rate which fills up the disk and crashes gluster.  I think gluster started to back up after awhile because I couldn't delete the files fast enough.  

Expected results:
I expected to see files in /var/lib/glusterd/stats/ that don't change in size and can be cat'd to find the latest stats.  

Additional info:
This is the patch that changed the behavior: https://github.com/gluster/glusterfs/commit/fc73ae5f81ef5926e3dc2311db116250d0f2a321.  Could you please revert it and backport it to the 3.12 series?

Comment 1 Chris Holcombe 2017-11-15 19:32:29 UTC
I added a revert change to review.gluster.org: https://review.gluster.org/#/c/18788/

Comment 2 Worker Ant 2017-11-17 22:36:05 UTC
REVIEW: https://review.gluster.org/18788 (Revert \"debug/io-stats: Append stats for each interval in the same file\") posted (#3) for review on master by

Comment 3 Worker Ant 2017-11-20 18:34:20 UTC
COMMIT: https://review.gluster.org/18788 committed in master by  with a commit message- Revert "debug/io-stats: Append stats for each interval in the same file"

This reverts commit fc73ae5f81ef5926e3dc2311db116250d0f2a321.
See bug: https://bugzilla.redhat.com/show_bug.cgi?id=1513692

Change-Id: I00d5989b042d4e345621c596d5370d324948557f
Bug: 1513692
Signed-off-by: Chris Holcombe <xfactor973@gmail.com>

Comment 4 Shyamsundar 2018-03-15 11:20:54 UTC
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-4.0.0, please open a new bug report.

glusterfs-4.0.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://lists.gluster.org/pipermail/announce/2018-March/000092.html
[2] https://www.gluster.org/pipermail/gluster-users/


Note You need to log in before you can comment on or make changes to this bug.