Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 1513479 - [RFE] Logging should show when glusterfs pod is killed due to node out of disk space
Summary: [RFE] Logging should show when glusterfs pod is killed due to node out of dis...
Alias: None
Product: Red Hat Gluster Storage
Classification: Red Hat
Component: core
Version: rhgs-3.4
Hardware: Unspecified
OS: Unspecified
Target Milestone: ---
: ---
Assignee: Sridhar Seshasayee
QA Contact: Rahul Hinduja
Depends On:
Blocks: 1476780 1542093
TreeView+ depends on / blocked
Reported: 2017-11-15 14:02 UTC by Raghavendra Talur
Modified: 2018-11-19 05:07 UTC (History)
10 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of: 1476780
Last Closed: 2018-11-19 05:07:22 UTC
Target Upstream Version:

Attachments (Terms of Use)

Comment 2 Raghavendra Talur 2017-11-15 14:12:52 UTC
The ask from user is that we log *somewhere* if glusterd/glusterfs/glusterfsd processes go down due to unavailability of space in /var/log or /var/lib.

I know this is chicken and egg kind of thing. Filing a bug to see if anyone has a intelligent way to fix it.

Comment 5 Amar Tumballi 2018-02-06 05:37:06 UTC
Having a centralized logging system, and a proper log-rotate functionality will resolve this issue IMO. Will keep this open for now and will plan to work on a recommendations to get over this issue.

Comment 8 Amar Tumballi 2018-10-17 10:57:25 UTC
No progress was made on this, AFAIK! Idea was to resolve it using centralized logging, which will have an event!

This has to be done ideally by monitoring, and monitoring /var/log/ is technically outside the scope of glusterfs. Also we have seen issues where a log (fprintf()) call can get hung when /var/log is full, so we can't implement it as a way to detect /var/log is full. Also, when we get a ENOSPC to a logging function, then we can never add any more log lines, thus, it is very much a monitoring problem!

That is the only best way to handle this. For storage hosted inside of glusterfs, we have got reserve space option, which helps us to reserve some space, which can be used to prevent issues, and also, that provides relevant logs!

Comment 9 Amar Tumballi 2018-11-19 05:07:22 UTC
Considering our new logrotate setting considers size as a parameter (from Batch Update 2, planned to be released next month). -

This bug should be a big issue with that feature, and with that should be fine to close it.

Will update the status again after the release.

Note You need to log in before you can comment on or make changes to this bug.