Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 1510573 - [DOCKER] Can't rsh into pod
Summary: [DOCKER] Can't rsh into pod
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Containers
Version: 3.6.0
Hardware: Unspecified
OS: Unspecified
unspecified
medium
Target Milestone: ---
: 3.9.0
Assignee: Lokesh Mandvekar
QA Contact: DeShuai Ma
URL:
Whiteboard:
: 1599756 (view as bug list)
Depends On:
Blocks: 1542093
TreeView+ depends on / blocked
 
Reported: 2017-11-07 17:10 UTC by Thom Carlin
Modified: 2018-07-18 14:39 UTC (History)
11 users (show)

Fixed In Version: docker-1.13.1-53.git774336d.el7_4
Doc Type: Bug Fix
Doc Text:
Cause: If containerA shares namespace, say ipc namespace, with containerB, then its ipc namespace path would be the same as containerB and be stored in state.json. Exec into containerA will just read the namespace paths stored in this file and join these namespaces. So, if containerB has already been stopped, docker exec containerA will fail. Consequence: docker exec containerA will fail. Fix: To address this issue, we should always save own namespace paths no matter if we share namespaces with other containers. Result: docker exec containerA will work.
Clone Of:
Environment:
Last Closed: 2018-03-28 14:11:22 UTC


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2018:0489 None None None 2018-03-28 14:11:45 UTC

Description Thom Carlin 2017-11-07 17:10:46 UTC
Description of problem:

oc rsh <<pod_name>> worked but commands hung for this pod.  Eventually, receive error message rsh'ing into pod

Version-Release number of selected component (if applicable):

3.6

How reproducible:

100% for this pod, other pod tested worked

Steps to Reproduce:
1. oc rsh <<pod_name>>

Actual results:

rpc error: code = 13 desc = invalid header field value "oci runtime error: exec failed: container_linux.go:247: starting container process caused \"process_linux.go:83: executing setns process caused \\\"exit status 16\\\"\"\n"
command terminated with exit code 126

Expected results:

Command prompt

Additional info:

CNS-backed pod running Coolstore MSA (gogs-postgresql)

From node /var/log/messages:
level=error msg="Error running exec in container: rpc error: code = 13 desc = invalid header field value \"oci runtime error: exec failed: container_linux.go:240: creating new parent process caused \\\"container_linux.go:1248: running lstat on namespace path \\\\\\\"/proc/94175/ns/ipc\\\\\\\" caused \\\\\\\"lstat /proc/94175/ns/ipc: no such file or directory\\\\\\\"\\\"\\n\""

W1107 12:09:22.963147   22572 prober.go:98] No ref for container "docker://1e61d150672b7737252f51834d71891c26fae394d63bf57fd60be8d9d74fa249" (gogs-postgresql-6-0dd1x_ci-ocuser(65ea0067-c3dc-11e7-92cf-001a4a160152):postgresql)

prober.go:106] Readiness probe for "gogs-postgresql-6-0dd1x_ci-ocuser(65ea0067-c3dc-11e7-92cf-001a4a160152):postgresql" failed (failure): rpc error: code = 13 desc = invalid header field value "oci runtime error: exec failed: container_linux.go:240: creating new parent process caused \"container_linux.go:1248: running lstat on namespace path \\\"/proc/94175/ns/ipc\\\" caused \\\"lstat /proc/94175/ns/ipc: no such file or directory\\\"\"\n"

Comment 1 Juan Vallejo 2017-11-07 21:37:04 UTC
Possible related upstream issue: https://github.com/opencontainers/runc/issues/1476

Comment 2 Antonio Murdaca 2017-11-12 14:24:17 UTC
Patch backported to:

- projectatomic/runc @ docker-1.12.6
- projectatomic/runc @ docker-1.13.1-rhel

Lokesh, pls make sure to rebuid docker with latest commits from the above branches.

Comment 4 DeShuai Ma 2018-01-04 09:18:41 UTC
Is there is step I can follow to verify the bug? I don't know how to reproduce the bug. thanks

Comment 5 DeShuai Ma 2018-01-04 09:38:51 UTC
which docker version fix the issue?

I follow the step in https://github.com/opencontainers/runc/issues/1476 still have the same failure

[root@ip-172-18-10-184 netns]# docker run -tid --name test000 ubuntu
916e495f4d2ad6459000655d8c19b14adf8228530c12185946897f87e6fe2f2d
[root@ip-172-18-10-184 netns]# docker run -tid --name test111 --ipc container:test000 ubuntu
bee80c5c44d86611f9283240cb29627ab39936e31d1fc408c21624c8f3de1e03
[root@ip-172-18-10-184 netns]# docker stop test000
test000
[root@ip-172-18-10-184 netns]# docker exec -ti test111 bash
rpc error: code = 13 desc = invalid header field value "oci runtime error: exec failed: container_linux.go:240: creating new parent process caused \"container_linux.go:1248: running lstat on namespace path \\\"/proc/40504/ns/ipc\\\" caused \\\"lstat /proc/40504/ns/ipc: no such file or directory\\\"\"\n"
[root@ip-172-18-10-184 netns]# 
[root@ip-172-18-10-184 netns]# docker version
Client:
 Version:         1.12.6
 API version:     1.24
 Package version: docker-1.12.6-68.gitec8512b.el7.x86_64
 Go version:      go1.8.3
 Git commit:      ec8512b/1.12.6
 Built:           Thu Nov 16 15:19:17 2017
 OS/Arch:         linux/amd64

Server:
 Version:         1.12.6
 API version:     1.24
 Package version: docker-1.12.6-68.gitec8512b.el7.x86_64
 Go version:      go1.8.3
 Git commit:      ec8512b/1.12.6
 Built:           Thu Nov 16 15:19:17 2017
 OS/Arch:         linux/amd64

Comment 8 Antonio Murdaca 2018-02-08 15:48:35 UTC
This is fixed in latest RPMs built from dist-git docker for 7.4 afaict.

 Package version: docker-1.12.6-71.git3e8e77d.el7.x86_64

^^ that's an old version from December actually, please test the latest package.

Assigning this to Lokesh, code wise, we have the patches in runc

Comment 11 DeShuai Ma 2018-02-23 03:07:05 UTC
Verify on docker-1.13.1-53.git774336d.el7.x86_64
# docker version
Client:
 Version:         1.13.1
 API version:     1.26
 Package version: <unknown>
 Go version:      go1.8.3
 Git commit:      774336d/1.13.1
 Built:           Tue Feb 20 13:46:34 2018
 OS/Arch:         linux/amd64

Server:
 Version:         1.13.1
 API version:     1.26 (minimum version 1.12)
 Package version: <unknown>
 Go version:      go1.8.3
 Git commit:      774336d/1.13.1
 Built:           Tue Feb 20 13:46:34 2018
 OS/Arch:         linux/amd64
 Experimental:    false
# docker run -tid --name test000 ubuntu
Unable to find image 'ubuntu:latest' locally
Trying to pull repository docker.io/library/ubuntu ... 
latest: Pulling from docker.io/library/ubuntu
1be7f2b886e8: Pull complete 
6fbc4a21b806: Pull complete 
c71a6f8e1378: Pull complete 
4be3072e5a37: Pull complete 
06c6d2f59700: Pull complete 
Digest: sha256:e27e9d7f7f28d67aa9e2d7540bdc2b33254b452ee8e60f388875e5b7d9b2b696
Status: Downloaded newer image for docker.io/ubuntu:latest
b37edd6140b6e768fca995f8b6a57440935c66b875196926567a3338a7792669

# docker run -tid --name test111 --ipc container:test000 ubuntu
2e55e3a29218035c60e53e74d5f4101e832202317e209f01473d78adfefb621a
# docker stop test000
test000
# docker exec -ti test111 bash
root@2e55e3a29218:/#

Comment 14 errata-xmlrpc 2018-03-28 14:11:22 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:0489

Comment 16 Derrick Ornelas 2018-07-18 14:39:24 UTC
*** Bug 1599756 has been marked as a duplicate of this bug. ***


Note You need to log in before you can comment on or make changes to this bug.