Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 1601809 - Mysql pod crash for Plugin 'InnoDB' init function returned error when use ceph RBD as default storageclass
Summary: Mysql pod crash for Plugin 'InnoDB' init function returned error when use cep...
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: Red Hat Software Collections
Classification: Red Hat
Component: rh-mysql57-container
Version: rh-mysql57
Hardware: Unspecified
OS: Unspecified
medium
high
Target Milestone: ---
: 3.3
Assignee: Jakub Jančo
QA Contact: qe-baseos-daemons
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-07-17 09:38 UTC by Wenjing Zheng
Modified: 2019-02-28 18:52 UTC (History)
9 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-02-28 18:52:33 UTC
Target Upstream Version:


Attachments (Terms of Use)

Description Wenjing Zheng 2018-07-17 09:38:41 UTC
Description of problem:
Met below error when test mysql-persistent template with ceph RBD storageclass:
2018-07-17T07:00:58.649670Z 0 [Note] InnoDB: Initializing buffer pool, total size = 256M, instances = 1, chunk size = 128M
2018-07-17T07:00:58.673477Z 0 [Note] InnoDB: Completed initialization of buffer pool
2018-07-17T07:00:58.677813Z 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
2018-07-17T07:00:58.688016Z 0 [ERROR] InnoDB: Header page consists of zero bytes in datafile: ./ibdata1, Space ID:0, Flags: 0. Please refer to http://dev.mysql.com/doc/refman/5.7/en/innodb-troubleshooting-datadict.html for how to resolve the issue.
2018-07-17T07:00:58.688060Z 0 [ERROR] InnoDB: Corrupted page [page id: space=0, page number=0] of datafile './ibdata1' could not be found in the doublewrite buffer.
2018-07-17T07:00:58.688077Z 0 [ERROR] InnoDB: Plugin initialization aborted with error Data structure corruption
2018-07-17T07:00:59.294636Z 0 [ERROR] Plugin 'InnoDB' init function returned error.
2018-07-17T07:00:59.294680Z 0 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
2018-07-17T07:00:59.294689Z 0 [ERROR] Failed to initialize builtin plugins.
2018-07-17T07:00:59.294693Z 0 [ERROR] Aborting

2018-07-17T07:00:59.294703Z 0 [Note] Binlog end
2018-07-17T07:00:59.294796Z 0 [Note] Shutting down plugin 'CSV'
2018-07-17T07:00:59.294808Z 0 [Note] Shutting down plugin 'MyISAM'
2018-07-17T07:00:59.297000Z 0 [Note] /opt/rh/rh-mysql57/root/usr/libexec/mysqld: Shutdown complete

Version-Release number of selected component (if applicable):
openshift v3.10.18

How reproducible:
always

Steps to Reproduce:
1.oc new-app --template=mysql-persistent
2.
3.

Actual results:
$ oc get pods
NAME             READY     STATUS             RESTARTS   AGE
mysql-1-deploy   0/1       Error              0          23m
mysql-2-deploy   1/1       Running            0          9m
mysql-2-gxlw9    0/1       CrashLoopBackOff   6          9m


Expected results:
Pods should be running like other persistent template.

Additional info:

Comment 5 Jakub Jančo 2019-02-12 15:13:12 UTC
Looking on this, currently Im not able to reproduce it. Could you please send complete reproducer code?

Comment 6 Jakub Jančo 2019-02-14 16:40:59 UTC
Please try these steps on testing instance:

1. stop/disable deployment of mysql
2. remove all database data from ceph RBD storage
3. in mysql deployment Config raise failureThreshold of livenessProbe and readinessProbe. By default it is 3, set at least 20.
4. start deployment

Comment 7 Josh Foots 2019-02-28 18:52:33 UTC
(In reply to Jakub Jančo from comment #6)
> Please try these steps on testing instance:
> 
> 1. stop/disable deployment of mysql
> 2. remove all database data from ceph RBD storage
> 3. in mysql deployment Config raise failureThreshold of livenessProbe and
> readinessProbe. By default it is 3, set at least 20.
> 4. start deployment

Provided customer with steps and customer confirmed that resolved it. I'll let you determine what to close the bug as. Marking as Closed NOTABUG and will create KCS incase someone else runs into this.

Thanks for all the assistance.

Cheers,
Josh


Note You need to log in before you can comment on or make changes to this bug.