Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 1356065 - data loss after rhgs node reboot
Summary: data loss after rhgs node reboot
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: Red Hat Gluster Storage
Classification: Red Hat
Component: heketi
Version: rhgs-3.1
Hardware: Unspecified
OS: Unspecified
urgent
high
Target Milestone: ---
: ---
Assignee: Humble Chirammal
QA Contact: Anoop
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-07-13 10:58 UTC by Neha
Modified: 2016-08-15 02:09 UTC (History)
8 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2016-07-15 07:16:14 UTC


Attachments (Terms of Use)

Description Neha 2016-07-13 10:58:59 UTC
Rebooted two of the rhgs node, now dont see heketi.db file under heketidb volume

due to that heketi container is crashing.


docker logs -f 87fcbdb9da33
Heketi 2.0.2
[heketi] INFO 2016/07/13 06:43:13 Loaded kubernetes executor
[heketi] ERROR 2016/07/13 06:43:13 /src/github.com/heketi/heketi/apps/glusterfs/app.go:149: write /var/lib/heketi/heketi.db: read-only file system
ERROR: Unable to start application


heketi container status : CrashLoopBackOff

Comment 3 Luis Pabón 2016-07-13 12:47:26 UTC
Humble, could you please investigate why a volume would go Read Only?

Comment 4 Humble Chirammal 2016-07-14 18:49:16 UTC
(In reply to Luis Pabón from comment #3)
> Humble, could you please investigate why a volume would go Read Only?

Luis, below error message clearly says 'The FS is READONLY' which is different from 'No Such File or Directory' reported here in the bugzilla. Can you confirm in this error scenario,  the path exist, but its READONLY causing this issue? 


[heketi] INFO 2016/07/13 06:43:13 Loaded kubernetes executor
[heketi] ERROR 2016/07/13 06:43:13 /src/github.com/heketi/heketi/apps/glusterfs/app.go:149: write /var/lib/heketi/heketi.db: read-only file system
ERROR: Unable to start application

Comment 5 Humble Chirammal 2016-07-14 18:52:17 UTC
@Neha, once again FS goes readonly in this scenario is expected as mentioned here https://bugzilla.redhat.com/show_bug.cgi?id=1355689. 

I dont know why you say the DB file *does not* exist eventhough there is a clear error message says FS is READONLY. I am confused with the bug report and the error message captured here in this bz as it does not match.

Comment 8 Neha 2016-07-15 05:26:42 UTC
Regarding read only volume , it might be because few of the brick process are not up and not because of data loss.

Comment 9 Neha 2016-07-15 07:16:14 UTC
closing this bug as heketi issue is being tracked in BZ 1355689. For data loss I will create a new BZ to avoid confusion.


Note You need to log in before you can comment on or make changes to this bug.