Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 1067008 - [new relic] Unable to delete application due to cartridge ident not found
Summary: [new relic] Unable to delete application due to cartridge ident not found
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: OpenShift Online
Classification: Red Hat
Component: Containers
Version: 1.x
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: ---
: ---
Assignee: Jhon Honce
QA Contact: libra bugs
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2014-02-19 14:01 UTC by Jessica Forrester
Modified: 2014-04-24 21:34 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2014-04-24 21:34:55 UTC
Target Upstream Version:


Attachments (Terms of Use)

Description Jessica Forrester 2014-02-19 14:01:34 UTC
Description of problem:
Exception reported from ApplicationsController#destroy for Cartridge Ident not found in /var/lib/openshift/<gear_id>/cdk

Prevented the application from being destroyed

Version-Release number of selected component (if applicable):
In PROD

Comment 2 Meng Bo 2014-03-13 11:21:13 UTC
Checked on devenv_4495, the issue still can be reproduced.


$ rhc app delete php53 
Deleting application 'php53' ... 
Unable to complete the requested operation due to: An invalid exit code (1) was returned from the server ip-10-113-144-174.  This indicates an unexpected problem
during the execution of your request.


Error in platform.log

March 13 07:15:55 WARN Failure while deleting gear 5321879163dd9e7c67000051: Cartridge Ident not found in /var/lib/openshift/5321879163dd9e7c67000051/10gen-mms-agent
March 13 07:15:55 INFO Failure while deleting gear 5321879163dd9e7c67000051: Cartridge Ident not found in /var/lib/openshift/5321879163dd9e7c67000051/10gen-mms-agent
/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.22.0/lib/openshift-origin-node/model/v2_cart_model.rb:174:in `get_cartridge_from_directory'
/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.22.0/lib/openshift-origin-node/model/v2_cart_model.rb:79:in `block in each_cartridge'
/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.22.0/lib/openshift-origin-node/model/v2_cart_model.rb:1128:in `block in process_cartridges'
/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.22.0/lib/openshift-origin-node/model/v2_cart_model.rb:1126:in `each'
/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.22.0/lib/openshift-origin-node/model/v2_cart_model.rb:1126:in `process_cartridges'
/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.22.0/lib/openshift-origin-node/model/v2_cart_model.rb:78:in `each_cartridge'
/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.22.0/lib/openshift-origin-node/model/application_container.rb:253:in `block in destroy'
/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-common-1.21.0/lib/openshift-origin-common/utils/path_utils.rb:93:in `block in flock'
/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-common-1.21.0/lib/openshift-origin-common/utils/file_needs_sync.rb:38:in `block in open'
/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-common-1.21.0/lib/openshift-origin-common/utils/file_needs_sync.rb:36:in `open'
/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-common-1.21.0/lib/openshift-origin-common/utils/file_needs_sync.rb:36:in `open'
/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-common-1.21.0/lib/openshift-origin-common/utils/path_utils.rb:88:in `flock'
/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.22.0/lib/openshift-origin-node/model/application_container.rb:251:in `destroy'
/opt/rh/ruby193/root/usr/libexec/mcollective/mcollective/agent/openshift.rb:373:in `oo_app_destroy'
/opt/rh/ruby193/root/usr/libexec/mcollective/mcollective/agent/openshift.rb:166:in `execute_action'
/opt/rh/ruby193/root/usr/libexec/mcollective/mcollective/agent/openshift.rb:134:in `cartridge_do_action'
/opt/rh/ruby193/root/usr/share/ruby/mcollective/rpc/agent.rb:86:in `handlemsg'
/opt/rh/ruby193/root/usr/share/ruby/mcollective/agents.rb:126:in `block (2 levels) in dispatch'
/opt/rh/ruby193/root/usr/share/ruby/timeout.rb:69:in `timeout'
/opt/rh/ruby193/root/usr/share/ruby/mcollective/agents.rb:125:in `block in dispatch'

Comment 3 Jhon Honce 2014-03-13 15:47:06 UTC
Fixed in https://github.com/openshift/origin-server/pull/4949
Image: devenv_4503

Comment 4 Meng Bo 2014-03-14 02:55:52 UTC
Checked on devenv_4509, the gear without IDENT env cannot be deleted. But the output is still same as comment#2.

# rhc app delete php54
Deleting application 'php54' ...
Unable to complete the requested operation due to: An invalid exit code (1) was returned from the server ip-10-69-147-203.  This indicates an unexpected problem
during the execution of your request.
Reference ID: e9693bdf0d2f8a0eec38f45820bb967d


March 13 22:52:38 WARN Failure while deleting gear 53226c2fb7668fb341000006: Cartridge Ident not found in /var/lib/openshift/53226c2fb7668fb341000006/php
March 13 22:52:38 INFO Failure while deleting gear 53226c2fb7668fb341000006: Cartridge Ident not found in /var/lib/openshift/53226c2fb7668fb341000006/php
/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.22.0/lib/openshift-origin-node/model/v2_cart_model.rb:174:in `get_cartridge_from_directory'
/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.22.0/lib/openshift-origin-node/model/v2_cart_model.rb:79:in `block in each_cartridge'
/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.22.0/lib/openshift-origin-node/model/v2_cart_model.rb:1128:in `block in process_cartridges'
/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.22.0/lib/openshift-origin-node/model/v2_cart_model.rb:1126:in `each'
/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.22.0/lib/openshift-origin-node/model/v2_cart_model.rb:1126:in `process_cartridges'
/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.22.0/lib/openshift-origin-node/model/v2_cart_model.rb:78:in `each_cartridge'
/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.22.0/lib/openshift-origin-node/model/application_container.rb:253:in `block in destroy'
/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-common-1.21.0/lib/openshift-origin-common/utils/path_utils.rb:93:in `block in flock'
/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-common-1.21.0/lib/openshift-origin-common/utils/file_needs_sync.rb:38:in `block in open'
/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-common-1.21.0/lib/openshift-origin-common/utils/file_needs_sync.rb:36:in `open'
/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-common-1.21.0/lib/openshift-origin-common/utils/file_needs_sync.rb:36:in `open'
/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-common-1.21.0/lib/openshift-origin-common/utils/path_utils.rb:88:in `flock'
/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.22.0/lib/openshift-origin-node/model/application_container.rb:251:in `destroy'
/opt/rh/ruby193/root/usr/libexec/mcollective/mcollective/agent/openshift.rb:373:in `oo_app_destroy'
/opt/rh/ruby193/root/usr/libexec/mcollective/mcollective/agent/openshift.rb:166:in `execute_action'
/opt/rh/ruby193/root/usr/libexec/mcollective/mcollective/agent/openshift.rb:134:in `cartridge_do_action'
/opt/rh/ruby193/root/usr/share/ruby/mcollective/rpc/agent.rb:86:in `handlemsg'
/opt/rh/ruby193/root/usr/share/ruby/mcollective/agents.rb:126:in `block (2 levels) in dispatch'
/opt/rh/ruby193/root/usr/share/ruby/timeout.rb:69:in `timeout'
/opt/rh/ruby193/root/usr/share/ruby/mcollective/agents.rb:125:in `block in dispatch'

Comment 5 Jhon Honce 2014-03-14 16:55:54 UTC
$ rhc create-app php54 php-5.4
...
$ rm /var/lib/openshift/*/php/env/*IDENT
...
$ rhc delete-app php54 --confirm
Deleting application 'php54' ... deleted

Fixed in https://github.com/openshift/origin-server/pull/4961

Comment 6 openshift-github-bot 2014-03-14 18:23:08 UTC
Commit pushed to master at https://github.com/openshift/origin-server

https://github.com/openshift/origin-server/commit/fd078a3ee80bdb332364a7205ba1573d0d393863
Bug 1067008 - Delete gear when missing a Cartridge Ident

Comment 7 Meng Bo 2014-03-17 02:34:50 UTC
Checked on devenv_4522, gear with IDENT env missing can be deleted now.

[root@fedora20 openshifttest]# rhc app delete php54
Deleting application 'php54' ... deleted

Following messages found in platform.log:
March 16 22:29:16 INFO openshift-agent: request start: action=cartridge_do requestid=25fdaab6ab3f59259047b1a
d595b6e01, senderid=ip-10-69-148-224, data={:cartridge=>"openshift-origin-node", :action=>"app-destroy", :ar
gs=>{"--with-app-uuid"=>"53265c358a16cb0a60000008", "--with-app-name"=>"php54", "--with-container-uuid"=>"53
265c358a16cb0a60000008", "--with-container-name"=>"php54", "--with-namespace"=>"bmengdev", "--with-request-i
d"=>"867db7b26b4dd0045649b1af7377a8ff", "--cart-name"=>"openshift-origin-node"}, :process_results=>true}
March 16 22:29:16 WARN Failure while deleting gear 53265c358a16cb0a60000008: Cartridge Ident not found in /v
ar/lib/openshift/53265c358a16cb0a60000008/php
March 16 22:29:16 INFO Failure while deleting gear 53265c358a16cb0a60000008: Cartridge Ident not found in /v
ar/lib/openshift/53265c358a16cb0a60000008/php


After gear deleted, do oo-accept-node to check, following mis-match found.

FAIL: httpd config references UUID without associated gear: '53265c358a16cb0a60000008'

Not sure if the residual gear configs are intended. Please have a look.

Comment 8 Jhon Honce 2014-03-17 16:35:54 UTC
There are operational scripts to correct the stale front end so we won't be duplicating that code here.

This style of delete should only happen in rare instances, for example where a previous delete or deconfigure has failed.

Comment 9 Meng Bo 2014-03-18 02:17:50 UTC
According to comment#7 and comment#8, move the bug to verified.


Note You need to log in before you can comment on or make changes to this bug.