Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 1063728 - [origin_node_324] Should close the listening ports for the gear after deleted via oo-admin-gear
Summary: [origin_node_324] Should close the listening ports for the gear after deleted...
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: OpenShift Online
Classification: Red Hat
Component: Containers
Version: 2.x
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: ---
: ---
Assignee: Lokesh Mandvekar
QA Contact: libra bugs
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2014-02-11 10:50 UTC by Meng Bo
Modified: 2015-05-14 23:34 UTC (History)
2 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2014-02-26 19:12:21 UTC
Target Upstream Version:


Attachments (Terms of Use)

Description Meng Bo 2014-02-11 10:50:44 UTC
Description of problem:
Delete the existing gear via oo-admin-gear -a destroygear, the listening port of the gear is still open. This may make the following gear creating fail.

Creating application 'perl2' ... 
Unexpected error: Failed to create the following private endpoints due to existing process bindings: OPENSHIFT_PERL_IP(8080)=127.1.244.1;

Checking the port of the ip:

# /usr/sbin/lsof -i @127.1.244.1:8080
lsof: no pwd entry for UID 1000
lsof: no pwd entry for UID 1000
COMMAND  PID     USER   FD   TYPE  DEVICE SIZE/OFF NODE NAME
lsof: no pwd entry for UID 1000
httpd   4896     1000    4u  IPv4 2176965      0t0  TCP 127.1.244.1:webcache (LISTEN)
lsof: no pwd entry for UID 1000
httpd   4900     1000    4u  IPv4 2176965      0t0  TCP 127.1.244.1:webcache (LISTEN)

# netstat -vnat |grep 244.1
tcp        0      0 127.1.244.1:8080            0.0.0.0:*                   LISTEN      

The port is still in LISTEN status.

Version-Release number of selected component (if applicable):
devenv_4358

How reproducible:
always

Steps to Reproduce:
1. Create app
2. Delete the app via oo-admin-gear
# oo-admin-gear -a destroygear -u 52f9fc60a034a11bca0001cc
3. Create another app

Actual results:
Will meet the following error.
Creating application 'perl2' ... 
Unexpected error: Failed to create the following private endpoints due to existing process bindings: OPENSHIFT_PERL_IP(8080)=127.1.244.1;

Expected results:
The app should be created successfully.

Additional info:
error log in platform.log:

February 11 05:31:53 INFO Creating cartridge directory 52f9fc12a034a11bca0001ba/perl
February 11 05:31:53 INFO Shell command '/bin/cp -ad /var/lib/openshift/.cartridge_repository/redhat-perl/0.0.9/versions /var/lib/openshift/.cartridge_repository/redhat-perl/0.0.9/run /var/lib/openshift/.cartridge_repository/redhat-perl/0.0.9/bin /var/lib/openshift/.cartridge_repository/redhat-perl/0.0.9/metadata /var/lib/openshift/.cartridge_repository/redhat-perl/0.0.9/README.md /var/lib/openshift/.cartridge_repository/redhat-perl/0.0.9/logs /var/lib/openshift/.cartridge_repository/redhat-perl/0.0.9/COPYRIGHT /var/lib/openshift/.cartridge_repository/redhat-perl/0.0.9/LICENSE /var/lib/openshift/.cartridge_repository/redhat-perl/0.0.9/env /var/lib/openshift/52f9fc12a034a11bca0001ba/perl' ran. rc=0 out=
February 11 05:31:53 INFO Cartridge perl recorded as primary within gear 52f9fc12a034a11bca0001ba
February 11 05:31:54 INFO Created cartridge directory 52f9fc12a034a11bca0001ba/perl
February 11 05:31:54 INFO Creating 1 private endpoints for 52f9fc12a034a11bca0001ba/perl
February 11 05:31:54 INFO IPs already allocated for 8080 in gear 52f9fc12a034a11bca0001ba: []
February 11 05:31:54 INFO Created private endpoint for cart perl in gear 52f9fc12a034a11bca0001ba: [OPENSHIFT_PERL_IP=127.1.244.1, OPENSHIFT_PERL_PORT=8080]
February 11 05:31:54 INFO Shell command '/usr/sbin/lsof -i @127.1.244.1:8080' ran. rc=0 out=COMMAND  PID                     USER   FD   TYPE  DEVICE SIZE/OFF NODE NAME
httpd   1541 52f9fc12a034a11bca0001ba    4u  IPv4 2162906      0t0  TCP 127.1.244.1:webcache (LISTEN)
httpd   1545 52f9fc12a034a11bca0001ba    4u  IPv4 2162906      0t0  TCP 127.1.244.1:webcache (LISTEN)

February 11 05:31:54 INFO Shell command '/usr/sbin/lsof -sTCP:^CLOSE_WAIT,^FIN_WAIT1,^FIN_WAIT2 -i @127.1.244.1:8080' ran. rc=0 out=COMMAND  PID                     USER   FD   TYPE  DEVICE SIZE/OFF NODE NAME
httpd   1541 52f9fc12a034a11bca0001ba    4u  IPv4 2162906      0t0  TCP 127.1.244.1:webcache (LISTEN)
httpd   1545 52f9fc12a034a11bca0001ba    4u  IPv4 2162906      0t0  TCP 127.1.244.1:webcache (LISTEN)

February 11 05:31:54 ERROR Unexpected error during configure: Failed to create the following private endpoints due to existing process bindings: OPENSHIFT_PERL_IP(8080)=127.1.244.1; (RuntimeError)
  /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.20.2/lib/openshift-origin-node/model/v2_cart_model.rb:919:in `create_private_endpoints'
  /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.20.2/lib/openshift-origin-node/model/v2_cart_model.rb:269:in `block in configure'
  /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.20.2/lib/openshift-origin-node/utils/cgroups.rb:126:in `call'
  /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.20.2/lib/openshift-origin-node/utils/cgroups.rb:126:in `apply_profile'
  /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.20.2/lib/openshift-origin-node/utils/cgroups.rb:44:in `block (2 levels) in <class:Cgroups>'
  /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.20.2/lib/openshift-origin-node/model/v2_cart_model.rb:255:in `configure'
  /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.20.2/lib/openshift-origin-node/model/application_container_ext/cartridge_actions.rb:28:in `configure'
  /opt/rh/ruby193/root/usr/libexec/mcollective/mcollective/agent/openshift.rb:924:in `block in oo_configure'
  /opt/rh/ruby193/root/usr/libexec/mcollective/mcollective/agent/openshift.rb:322:in `with_container_from_args'
  /opt/rh/ruby193/root/usr/libexec/mcollective/mcollective/agent/openshift.rb:923:in `oo_configure'
  /opt/rh/ruby193/root/usr/libexec/mcollective/mcollective/agent/openshift.rb:149:in `execute_action'
  /opt/rh/ruby193/root/usr/libexec/mcollective/mcollective/agent/openshift.rb:101:in `cartridge_do_action'
  /opt/rh/ruby193/root/usr/share/ruby/mcollective/rpc/agent.rb:86:in `handlemsg'
  /opt/rh/ruby193/root/usr/share/ruby/mcollective/agents.rb:126:in `block (2 levels) in dispatch'
  /opt/rh/ruby193/root/usr/share/ruby/timeout.rb:69:in `timeout'
  /opt/rh/ruby193/root/usr/share/ruby/mcollective/agents.rb:125:in `block in dispatch'
February 11 05:31:54 INFO Shell command 'quota --always-resolve -w 52f9fc12a034a11bca0001ba' ran. rc=0 out=Disk quotas for user 52f9fc12a034a11bca0001ba (uid 1000): 
     Filesystem  blocks   quota   limit   grace   files   quota   limit   grace
     /dev/xvde2     216       0 1048576              58       0   80000        

February 11 05:31:54 INFO Deleting private endpoints for 52f9fc12a034a11bca0001ba/perl
February 11 05:31:54 INFO Deleted private endpoints for 52f9fc12a034a11bca0001ba/perl

Comment 1 Lokesh Mandvekar 2014-02-11 15:08:06 UTC
(In reply to Meng Bo from comment #0)
 
> Steps to Reproduce:
> 1. Create app
> 2. Delete the app via oo-admin-gear
> # oo-admin-gear -a destroygear -u 52f9fc60a034a11bca0001cc
> 3. Create another app
> 

Meng, just to confirm, AFAIK, oo-admin-gear would only be used when the app has already been deleted and there would still be stale gears/files left. I'm not sure if step 2 above would be performed without an rhc app delete. 

So, do we still see the problem occurring even when we include the app deletion step before using oo-admin-gear?

Jhon, correct me if I'm wrong here.

Comment 2 Lokesh Mandvekar 2014-02-11 18:30:12 UTC
https://github.com/openshift/origin-server/pull/4728

Comment 3 Meng Bo 2014-02-12 05:47:59 UTC
Checked on devenv_4363, issue has been fixed.

The following app creation will not fail due to the port is being used.

Move bug to verified.


Note You need to log in before you can comment on or make changes to this bug.