Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 1510653 - df displays "Transport endpoint is not connected" for CNS-backed volume
Summary: df displays "Transport endpoint is not connected" for CNS-backed volume
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Red Hat Gluster Storage
Classification: Red Hat
Component: kubernetes
Version: cns-3.6
Hardware: Unspecified
OS: Unspecified
unspecified
medium
Target Milestone: ---
: ---
Assignee: Humble Chirammal
QA Contact: Prasanth
URL:
Whiteboard:
Depends On:
Blocks: 1542093 1543779 1622458
TreeView+ depends on / blocked
 
Reported: 2017-11-07 22:25 UTC by Thom Carlin
Modified: 2018-11-06 09:11 UTC (History)
14 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-09-21 05:42:32 UTC


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Red Hat Knowledge Base (Solution) 3451531 None None None 2018-05-18 21:01:19 UTC

Description Thom Carlin 2017-11-07 22:25:40 UTC
Description of problem:

df command returns "var/lib/origin/openshift.local.volumes/pods/<<pod_uuid>>/volumes/kubernetes.io~glusterfs/pvc-<<pvc_uuid>>’: Transport endpoint is not connected" on single OCP node


Version-Release number of selected component (if applicable):

cns 3.6

How reproducible:

Uncertain

Steps to Reproduce:
1. On affected node: df
2. On affected node: ls -l /var/lib/origin/openshift.local.volumes/pods/<<pod_uuid>>/volumes/kubernetes.io~glusterfs/pvc-<<pvc_uuid>>
3.

Actual results:

/var/lib/origin/openshift.local.volumes/pods/<<pod_uuid>>/volumes/kubernetes.io~glusterfs/pvc-<<pvc_uuid>>’: Transport endpoint is not connected
ls: cannot access /var/lib/origin/openshift.local.volumes/pods<<pod_uuid>>/volumes/kubernetes.io~glusterfs/pvc-<<pvc_uuid>>: Transport endpoint is not connected

Expected results:

Normal df, ls output

Additional info:

grep pvc-<<pvc_uuid>> /var/log/messages returns:
E1107 17:23:07.040275  116096 glusterfs.go:145] glusterfs: failed to get endpoints pvc-<<pvc_uuid>>[an empty namespace may not be set when a resource name is provided]
E1107 17:23:07.040330  116096 reconciler.go:449] Could not construct volume information: MountVolume.NewMounter failed for volume "kubernetes.io/glusterfs/<<pod_uuid>>-pvc-<<pvc_uuid>>" (spec.Name: "pvc-<<pvc_uuid>>") pod "<<pod_uuid>>" (UID: "<<pod_uuid>>") with: an empty namespace may not be set when a resource name is provided

Comment 2 Thom Carlin 2017-11-07 22:29:23 UTC
Workaround: umount /var/lib/origin/openshift.local.volumes/pods/<<pod_uuid>>/volumes/kubernetes.io~glusterfs/pvc-<<pvc_uuid>>

This clears the erroneous mount and clears the message.  

Then: rmdir /var/lib/origin/openshift.local.volumes/pods/<<pod_uuid>>/volumes/kubernetes.io~glusterfs/pvc-<<pvc_uuid>>/

clears the pod and volume information

Comment 7 Thom Carlin 2018-05-18 19:50:32 UTC
More information:
oc delete project <<project-name>>

project changes status to Terminating

Looking at logs, it is waiting for pods to clear

oc get pods -o wide -n <<project-name>> lists the offending pods

running "df" on the involved node(s) results in the "Transport endpoint is not connected" error

The pods have PVCs dynamically provisioned by gluster

Comment 8 Thom Carlin 2018-05-18 19:56:06 UTC
Tests:
Are the endpoints created by the Gluster provisioner being deleted before the pods are?
A) try deleting all the pods (or Deployments, or whatever) first, then delete the project
B) If that works, try deleting the endpoints first then deleting the project.

Comment 9 Thom Carlin 2018-05-18 20:05:32 UTC
A) results:
Scaling each deploymentconfig to 0, THEN deleting the project worked

Comment 10 Thom Carlin 2018-05-18 23:05:50 UTC
B) Deleting the endpoints first, then deleting the project results in the original issue

Comment 12 Humble Chirammal 2018-09-21 05:42:32 UTC
(In reply to Thom Carlin from comment #7)
> More information:
> oc delete project <<project-name>>
> 
> project changes status to Terminating
> 
> Looking at logs, it is waiting for pods to clear
> 
> oc get pods -o wide -n <<project-name>> lists the offending pods
> 
> running "df" on the involved node(s) results in the "Transport endpoint is
> not connected" error
> 
> The pods have PVCs dynamically provisioned by gluster

iirc, this issue is fixed in >= OCP 3.7. I would like to close this issue for the same reason. If this is not fixed, please feel free to re-open the bug.


Note You need to log in before you can comment on or make changes to this bug.