Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 1607984 - [3.10][CRI-O] Cassandra doesn't start on crio
Summary: [3.10][CRI-O] Cassandra doesn't start on crio
Keywords:
Status: CLOSED WORKSFORME
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Hawkular
Version: 3.10.0
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: ---
: ---
Assignee: Antonio Murdaca
QA Contact: Junqi Zhao
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-07-24 16:34 UTC by Junqi Zhao
Modified: 2018-07-25 06:43 UTC (History)
11 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of: 1517652
Environment:
Last Closed: 2018-07-25 06:43:32 UTC


Attachments (Terms of Use)

Comment 1 Junqi Zhao 2018-07-24 16:39:41 UTC
The latest image with cri-o 1.10.5 is also have this issue, should seek help from cri-o team

metrics-cassandra-v3.10.23-1
metrics-hawkular-metrics-v3.10.23-1
metrics-schema-installer-v3.10.23-1
metrics-heapster-v3.10.23-1

Comment 2 Junqi Zhao 2018-07-25 02:16:35 UTC
Blocks metrics installation

Comment 3 Anping Li 2018-07-25 03:35:16 UTC
Both crictl exec and runc exec works well.

# crictl exec  7140ef4831fe7ced8d1f480ba3046f2f3dff8e85436dad015f7558821c882549 /opt/apache-cassandra/bin/cassandra-docker-ready.sh
W0724 23:33:33.423292   29209 util_unix.go:75] Using "/var/run/crio/crio.sock" as endpoint is deprecated, please consider using full url format "unix:///var/run/crio/crio.sock".
Picked up JAVA_TOOL_OPTIONS: -Duser.home=/home/jboss -Duser.name=jboss
Cassandra is in the up and normal state. It is now ready.


#runc exec 7140ef4831fe7ced8d1f480ba3046f2f3dff8e85436dad015f7558821c882549 /opt/apache-cassandra/bin/cassandra-docker-ready.sh
Picked up JAVA_TOOL_OPTIONS: -Duser.home=/home/jboss -Duser.name=jboss
Cassandra is in the up and normal state. It is now ready

Comment 4 Anping Li 2018-07-25 03:41:40 UTC
The interesting thing is that when the hawkular-cassandra pod is deleted, the new  pod could be ready.
so the workaround is to delete the hawkular-cassandra pod.

Comment 5 Junqi Zhao 2018-07-25 06:04:30 UTC
the default timeoutSeconds is 1 second, but the readiness check takes more than 1 second to get the response

# time oc rsh pod/hawkular-cassandra-1-prh6t /opt/apache-cassandra/bin/cassandra-docker-ready.sh
Picked up JAVA_TOOL_OPTIONS: -Duser.home=/home/jboss -Duser.name=jboss
Cassandra is in the up and normal state. It is now ready.

real	0m1.360s
user	0m0.140s
sys	0m0.041s


after changing it to bigger value in roles/openshift_metrics/templates/hawkular_cassandra_rc.j2, added timeoutSeconds: 10, metrics works well.

        readinessProbe:
          exec:
            command:
            - "/opt/apache-cassandra/bin/cassandra-docker-ready.sh"
          timeoutSeconds: 10

Comment 6 Junqi Zhao 2018-07-25 06:43:32 UTC
Per Comment 5, close this defect and filed Bug 1608216 to change the default timeoutSeconds for hawkular-cassandra pod's readiness check


Note You need to log in before you can comment on or make changes to this bug.