Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 1064013 - Random test case failures with accept-node
Summary: Random test case failures with accept-node
Keywords:
Status: CLOSED WONTFIX
Alias: None
Product: OpenShift Online
Classification: Red Hat
Component: Containers
Version: 2.x
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: ---
: ---
Assignee: Rajat Chopra
QA Contact: libra bugs
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2014-02-11 19:19 UTC by Dan McPherson
Modified: 2015-06-11 22:09 UTC (History)
0 users

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2015-06-11 22:09:26 UTC


Attachments (Terms of Use)

Description Dan McPherson 2014-02-11 19:19:25 UTC
Description of problem:

From:

https://ci.dev.openshift.redhat.com/jenkins/job/gear_extended/3/artifact/li/rhc/log/node/platform.log


February 11 10:46:51 INFO Shell command 'oo-iptables-port-proxy showproxy 38047 | awk '{ print $2 }'' ran. rc=0 out=
February 11 10:46:52 INFO Shell command 'oo-iptables-port-proxy addproxy 38047 "127.1.245.129:8080"' ran. rc=0 out=-I rhc-app-comm 1 -d 127.1.245.129 -p tcp --dport 8080 -j ACCEPT -m comment --comment 38047
-I rhc-app-comm 1 -d 127.1.245.129 -m conntrack --ctstate NEW -m tcp -p tcp --dport 8080 -j ACCEPT -m comment --comment 38047
-A OUTPUT -d 10.235.31.184/32 -m tcp -p tcp --dport 38047 -j DNAT --to-destination 127.1.245.129:8080
-A PREROUTING -d 10.235.31.184/32 -m tcp -p tcp --dport 38047 -j DNAT --to-destination 127.1.245.129:8080

February 11 10:46:52 INFO Created public endpoint for cart mock in gear 52fa45d1b3e342eb6700008c: [OPENSHIFT_MOCK_EXAMPLE_PUBLIC_PORT1=38047]
February 11 10:46:52 INFO Shell command 'oo-iptables-port-proxy showproxy 38046 | awk '{ print $2 }'' ran. rc=0 out=127.1.245.131:8443

February 11 10:46:52 INFO Shell command '/sbin/runuser -s /bin/sh 52fa452db3e34245b5000058 -c "exec /usr/bin/runcon 'unconfined_u:system_r:openshift_t:s0:c0,c1008' /bin/sh -c \"/var/lib/openshift/52fa452db3e34245b5000058/haproxy/hooks/set-haproxy-status-url app24276669 ns59997768 52fa452db3e34245b5000058 \'52fa452db3e34245b5000058\'\=\'http://app24276669-ns59997768.dev.rhcloud.com/haproxy-status/'
'\'\ \'52fa456fb3e3421e1d000006\'\=\'http://52fa456fb3e3421e1d000006-ns59997768.dev.rhcloud.com/haproxy-status/'
'\'\""' ran. rc=0 out=
February 11 10:46:52 INFO (0)
------

------)
February 11 10:46:52 INFO Shell command 'quota --always-resolve -w 52fa452db3e34245b5000058' ran. rc=0 out=Disk quotas for user 52fa452db3e34245b5000058 (uid 1008): 
     Filesystem  blocks   quota   limit   grace   files   quota   limit   grace
     /dev/xvde2     912       0 1048576             258       0   80000        

February 11 10:46:52 INFO Shell command 'oo-iptables-port-proxy showproxy 38047 | awk '{ print $2 }'' ran. rc=0 out=127.1.245.129:8080

February 11 10:46:52 INFO Shell command 'useradd -u 1006                     -d /var/lib/openshift/52fa45e9b3e342a58b0000ed                     -s /usr/bin/oo-trap-user                     -c 'OpenShift guest'                     -m                     -k /etc/openshift/skel                     52fa45e9b3e342a58b0000ed -G ""' ran. rc=0 out=
February 11 10:46:52 INFO Shell command 'oo-iptables-port-proxy showproxy 38048 | awk '{ print $2 }'' ran. rc=0 out=
February 11 10:46:53 INFO Shell command 'ip link show dev eth0' ran. rc=0 out=2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc htb state UP qlen 1000
    link/ether 22:00:0a:eb:1f:b8 brd ff:ff:ff:ff:ff:ff

February 11 10:46:53 INFO Shell command 'oo-iptables-port-proxy addproxy 38048 "127.1.245.129:8081"' ran. rc=0 out=-I rhc-app-comm 1 -d 127.1.245.129 -p tcp --dport 8081 -j ACCEPT -m comment --comment 38048
-I rhc-app-comm 1 -d 127.1.245.129 -m conntrack --ctstate NEW -m tcp -p tcp --dport 8081 -j ACCEPT -m comment --comment 38048
-A OUTPUT -d 10.235.31.184/32 -m tcp -p tcp --dport 38048 -j DNAT --to-destination 127.1.245.129:8081
-A PREROUTING -d 10.235.31.184/32 -m tcp -p tcp --dport 38048 -j DNAT --to-destination 127.1.245.129:8081

February 11 10:46:53 INFO Created public endpoint for cart mock in gear 52fa45d1b3e342eb6700008c: [OPENSHIFT_MOCK_EXAMPLE_PUBLIC_PORT2=38048]
February 11 10:46:53 INFO Shell command 'tc -s class show dev eth0 classid 1:3ec' ran. rc=0 out=
February 11 10:46:53 INFO Shell command '/sbin/runuser -s /bin/sh 52fa456fb3e3421e1d000006 -c "exec /usr/bin/runcon 'unconfined_u:system_r:openshift_t:s0:c0,c1000' /bin/sh -c \"/var/lib/openshift/52fa456fb3e3421e1d000006/haproxy/hooks/set-haproxy-status-url 52fa456fb3e3421e1d000006 ns59997768 52fa456fb3e3421e1d000006 \'52fa452db3e34245b5000058\'\=\'http://app24276669-ns59997768.dev.rhcloud.com/haproxy-status/'
'\'\ \'52fa456fb3e3421e1d000006\'\=\'http://52fa456fb3e3421e1d000006-ns59997768.dev.rhcloud.com/haproxy-status/'
'\'\""' ran. rc=0 out=
February 11 10:46:53 INFO (0)



Maps back to:

Node Acceptance Output = FAIL: user 888356962495462394298368 does not have a PAM limits file
FAIL: user 52fa45e9b3e342a58b0000ed does not have a PAM limits file
FAIL: directory 888356962495462394298368 doesn't have a cartridge directory
FAIL: directory 52fa45e9b3e342a58b0000ed doesn't have a .ssh directory
FAIL: directory 52fa45e9b3e342a58b0000ed doesn't have a .env directory
FAIL: directory 52fa45e9b3e342a58b0000ed doesn't have a .sandbox directory
FAIL: directory 52fa45e9b3e342a58b0000ed doesn't have a .tmp directory
FAIL: directory 52fa45e9b3e342a58b0000ed doesn't have a cartridge directory
FAIL: Gear does not have an OPENSHIFT_GEAR_DNS variable: '52fa45e9b3e342a58b0000ed'


Found in the result.


The details of the other id listed are also port proxy related it seems:

February 11 10:46:51 INFO Shell command 'oo-iptables-port-proxy showproxy 38047 | awk '{ print $2 }'' ran. rc=0 out=
February 11 10:46:52 INFO Shell command 'oo-iptables-port-proxy addproxy 38047 "127.1.245.129:8080"' ran. rc=0 out=-I rhc-app-comm 1 -d 127.1.245.129 -p tcp --dport 8080 -j ACCEPT -m comment --comment 38047
-I rhc-app-comm 1 -d 127.1.245.129 -m conntrack --ctstate NEW -m tcp -p tcp --dport 8080 -j ACCEPT -m comment --comment 38047
-A OUTPUT -d 10.235.31.184/32 -m tcp -p tcp --dport 38047 -j DNAT --to-destination 127.1.245.129:8080
-A PREROUTING -d 10.235.31.184/32 -m tcp -p tcp --dport 38047 -j DNAT --to-destination 127.1.245.129:8080

February 11 10:46:52 INFO Created public endpoint for cart mock in gear 52fa45d1b3e342eb6700008c: [OPENSHIFT_MOCK_EXAMPLE_PUBLIC_PORT1=38047]
February 11 10:46:52 INFO Shell command 'oo-iptables-port-proxy showproxy 38046 | awk '{ print $2 }'' ran. rc=0 out=127.1.245.131:8443

February 11 10:46:52 INFO Shell command '/sbin/runuser -s /bin/sh 52fa452db3e34245b5000058 -c "exec /usr/bin/runcon 'unconfined_u:system_r:openshift_t:s0:c0,c1008' /bin/sh -c \"/var/lib/openshift/52fa452db3e34245b5000058/haproxy/hooks/set-haproxy-status-url app24276669 ns59997768 52fa452db3e34245b5000058 \'52fa452db3e34245b5000058\'\=\'http://app24276669-ns59997768.dev.rhcloud.com/haproxy-status/'
'\'\ \'52fa456fb3e3421e1d000006\'\=\'http://52fa456fb3e3421e1d000006-ns59997768.dev.rhcloud.com/haproxy-status/'
'\'\""' ran. rc=0 out=
February 11 10:46:52 INFO (0)
------

------)
February 11 10:46:52 INFO Shell command 'quota --always-resolve -w 52fa452db3e34245b5000058' ran. rc=0 out=Disk quotas for user 52fa452db3e34245b5000058 (uid 1008): 
     Filesystem  blocks   quota   limit   grace   files   quota   limit   grace
     /dev/xvde2     912       0 1048576             258       0   80000        

February 11 10:46:52 INFO Shell command 'oo-iptables-port-proxy showproxy 38047 | awk '{ print $2 }'' ran. rc=0 out=127.1.245.129:8080

February 11 10:46:52 INFO Shell command 'useradd -u 1006                     -d /var/lib/openshift/52fa45e9b3e342a58b0000ed                     -s /usr/bin/oo-trap-user                     -c 'OpenShift guest'                     -m                     -k /etc/openshift/skel                     52fa45e9b3e342a58b0000ed -G ""' ran. rc=0 out=
February 11 10:46:52 INFO Shell command 'oo-iptables-port-proxy showproxy 38048 | awk '{ print $2 }'' ran. rc=0 out=
February 11 10:46:53 INFO Shell command 'ip link show dev eth0' ran. rc=0 out=2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc htb state UP qlen 1000
    link/ether 22:00:0a:eb:1f:b8 brd ff:ff:ff:ff:ff:ff

February 11 10:46:53 INFO Shell command 'oo-iptables-port-proxy addproxy 38048 "127.1.245.129:8081"' ran. rc=0 out=-I rhc-app-comm 1 -d 127.1.245.129 -p tcp --dport 8081 -j ACCEPT -m comment --comment 38048
-I rhc-app-comm 1 -d 127.1.245.129 -m conntrack --ctstate NEW -m tcp -p tcp --dport 8081 -j ACCEPT -m comment --comment 38048
-A OUTPUT -d 10.235.31.184/32 -m tcp -p tcp --dport 38048 -j DNAT --to-destination 127.1.245.129:8081
-A PREROUTING -d 10.235.31.184/32 -m tcp -p tcp --dport 38048 -j DNAT --to-destination 127.1.245.129:8081

February 11 10:46:53 INFO Created public endpoint for cart mock in gear 52fa45d1b3e342eb6700008c: [OPENSHIFT_MOCK_EXAMPLE_PUBLIC_PORT2=38048]
February 11 10:46:53 INFO Shell command 'tc -s class show dev eth0 classid 1:3ec' ran. rc=0 out=
February 11 10:46:53 INFO Shell command '/sbin/runuser -s /bin/sh 52fa456fb3e3421e1d000006 -c "exec /usr/bin/runcon 'unconfined_u:system_r:openshift_t:s0:c0,c1000' /bin/sh -c \"/var/lib/openshift/52fa456fb3e3421e1d000006/haproxy/hooks/set-haproxy-status-url 52fa456fb3e3421e1d000006 ns59997768 52fa456fb3e3421e1d000006 \'52fa452db3e34245b5000058\'\=\'http://app24276669-ns59997768.dev.rhcloud.com/haproxy-status/'
'\'\ \'52fa456fb3e3421e1d000006\'\=\'http://52fa456fb3e3421e1d000006-ns59997768.dev.rhcloud.com/haproxy-status/'
'\'\""' ran. rc=0 out=
February 11 10:46:53 INFO (0)

Comment 1 Mrunal Patel 2014-02-11 20:02:17 UTC
The .state is missing for one of the gear:

February 11 10:47:02 INFO Failed to get state: 52fa45e9b3e342a58b0000ed [/var/lib/openshift/52fa45e9b3e342a58b0000ed/app-root/runtime/.state]: No such file or directory
February 11 10:48:02 INFO Failed to get state: 52fa45e9b3e342a58b0000ed [/var/lib/openshift/52fa45e9b3e342a58b0000ed/app-root/runtime/.state]: No such file or directory
February 11 10:49:02 INFO Failed to get state: 52fa45e9b3e342a58b0000ed [/var/lib/openshift/52fa45e9b3e342a58b0000ed/app-root/runtime/.state]: No such file or directory
February 11 10:50:03 INFO Failed to get state: 52fa45e9b3e342a58b0000ed [/var/lib/openshift/52fa45e9b3e342a58b0000ed/app-root/runtime/.state]: No such file or directory
February 11 10:51:03 INFO Failed to get state: 52fa45e9b3e342a58b0000ed [/var/lib/openshift/52fa45e9b3e342a58b0000ed/app-root/runtime/.state]: No such file or directory
February 11 10:52:03 INFO Failed to get state: 52fa45e9b3e342a58b0000ed [/var/lib/openshift/52fa45e9b3e342a58b0000ed/app-root/runtime/.state]: No such file or directory

Comment 2 Mrunal Patel 2014-02-11 20:18:06 UTC
[mrunal@localhost log]$ grep showproxy node/platform.log | wc -l
628
[mrunal@localhost log]$ grep showproxy node/platform.log | grep "rc=0" | wc -l
628
[mrunal@localhost log]$ grep addproxy node/platform.log | grep "rc=0" | wc -l
231
[mrunal@localhost log]$ grep addproxy node/platform.log |  wc -l
231
[mrunal@localhost log]$ grep removeproxy node/platform.log |  wc -l
158
[mrunal@localhost log]$ grep removeproxy node/platform.log |  grep "rc=0" | wc -l
158

Comment 3 Rajat Chopra 2014-03-12 18:27:03 UTC
Cannot reproduce. See comment#2 as an example.

Comment 4 Dan McPherson 2014-03-12 18:43:50 UTC
It still happens all the time still.  Just look at merge/test/libra_ami/and extended test runs.

Comment 5 Dan McPherson 2014-04-08 18:09:40 UTC
An easy way to recreate this is break an external service like dns (change the server it points to to be invalid).  And run through the test cases.  Pretty consistently this will leave garbage around that accept-node will fail on.

Comment 6 Rajat Chopra 2014-04-12 00:30:39 UTC
One of the major causes is the failure to clean-up in some of the tests :
 https://github.com/openshift/origin-server/blob/master/console/test/functional/applications_controller_sanity_test.rb

Most of the console tests do not have a teardown method, so if any of the 'assert' statements fail with gear in an 'ugly' state, the leftover gear will cause oo-accept-node to fail.


Note You need to log in before you can comment on or make changes to this bug.