Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 1056480 - Failed to control app after quota exceeds 90%
Summary: Failed to control app after quota exceeds 90%
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: OpenShift Online
Classification: Red Hat
Component: Containers
Version: 2.x
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: ---
: ---
Assignee: Jhon Honce
QA Contact: libra bugs
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2014-01-22 09:42 UTC by chunchen
Modified: 2016-09-30 02:15 UTC (History)
1 user (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2014-02-26 19:09:59 UTC


Attachments (Terms of Use)
the thread-dump logs in development.log (deleted)
2014-01-22 09:42 UTC, chunchen
no flags Details

Description chunchen 2014-01-22 09:42:37 UTC
Created attachment 853763 [details]
the thread-dump logs in development.log

Description of problem:
It is failed to thread-dump after quota exceeds 90%, will meet "An invalid exit code (nil) was returned from the server ip-10-180-231-119.  This indicates an unexpected problem during the execution of your request." errors in development.log

Version-Release number of selected component (if applicable):
devenv_4257

How reproducible:
always

Steps to Reproduce:
1. Create an app which can be thread-dump like ruby-1.9
rhc app create crb19 ruby-1.9 --no-git

2. SSH into crb19
rhc ssh crb19

3. Consume 90+% space of the gear, e.g add a large file in app-root/data by "dd if=/dev/zero of=~/app-root/data/testfile bs=1M count=950"

4. Access the app webpage

5. Thread-dump this app
rhc threaddump crb19

Actual results:
return nothing

Expected results:
Thread-dump after quota exceeds 90% should be successfully.

Additional info:
1. Please see the attachment for the detail logs.
2. Tried ruby-1.9 and jbossas-7, they are all failed.

Comment 1 chunchen 2014-01-22 09:56:47 UTC
When operating app like stop/start/restart after quota exceeds 90%, they are also failed, all met "A gear stop did not complete on 1 gear. Please try again and contact support if the issue persists." errors

Comment 2 openshift-github-bot 2014-01-22 18:24:50 UTC
Commit pushed to master at https://github.com/openshift/origin-server

https://github.com/openshift/origin-server/commit/06baec9aefd6f29316b7a2ca6484dad331440653
Bug 1056480 - Removed random character in code

Comment 3 chunchen 2014-01-23 02:32:28 UTC
It's fixed, verified on devenv_4261, please refer to the following results:

1. Create an app
rhc app create crb19 ruby-1.9 --no-git

2. SSH into this app
rhc ssh crb19

3. Consume 90+% space of the gear, e.g add a large file in app-root/data by "dd if=/dev/zero of=~/app-root/data/testfile bs=1M count=950"

4. Access the app webpage

5. Control this app
1) $ rhc threaddump crb19
Warning: Gear 52e07c61cd9bea0aa7000007 is using 92.9% of disk quota
Success

The thread dump file will be available via: rhc tail crb19 -g 52e07c61cd9bea0aa7000007 -f /var/lib/openshift/52e07c61cd9bea0aa7000007/ruby//logs//error_log-20140123-* -o '-n 250'

2) $ rhc app restart crb19
Warning: Gear 52e07c61cd9bea0aa7000007 is using 92.9% of disk quota

RESULT:
crb19 restarted

3) $ rhc cartridge add mysql-5.1 -a crb19
Adding mysql-5.1 to application 'crb19' ... 
Warning: Gear 52e07c61cd9bea0aa7000007 is using 93.0% of disk quota
Warning: Gear 52e07c61cd9bea0aa7000007 is using 97.1% of disk quota
done

mysql-5.1 (MySQL 5.1)
---------------------
  Gears:          Located with ruby-1.9
  Connection URL: mysql://$OPENSHIFT_MYSQL_DB_HOST:$OPENSHIFT_MYSQL_DB_PORT/
  Database Name:  crb19
  Password:       7YCWP-uR27KI
  Username:       adminiBUfasB

MySQL 5.1 database added.  Please make note of these credentials:

       Root User: adminiBUfasB
   Root Password: 7YCWP-uR27KI
   Database Name: crb19

Connection URL: mysql://$OPENSHIFT_MYSQL_DB_HOST:$OPENSHIFT_MYSQL_DB_PORT/

You can manage your new MySQL database by also embedding phpmyadmin.
The phpmyadmin username and password will be the same as the MySQL credentials above.

4) $ rhc cartridge restart mysql-5.1 -a crb19
Restarting mysql-5.1 ... Warning: Gear 52e07c61cd9bea0aa7000007 is using 97.1% of disk quota
done


Note You need to log in before you can comment on or make changes to this bug.