Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 1055869 - Add mysql cartridge failed with 'ERROR 2013 (HY000)'
Summary: Add mysql cartridge failed with 'ERROR 2013 (HY000)'
Keywords:
Status: CLOSED INSUFFICIENT_DATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Image
Version: 2.0.0
Hardware: Unspecified
OS: Unspecified
unspecified
low
Target Milestone: ---
: ---
Assignee: Jason DeTiberus
QA Contact: libra bugs
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2014-01-21 06:19 UTC by Anping Li
Modified: 2014-02-04 19:04 UTC (History)
1 user (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2014-02-04 19:04:35 UTC
Target Upstream Version:


Attachments (Terms of Use)
Production log (deleted)
2014-01-21 06:22 UTC, Anping Li
no flags Details
Platform-trace log (deleted)
2014-01-21 06:23 UTC, Anping Li
no flags Details

Description Anping Li 2014-01-21 06:19:57 UTC
Description of problem:
This is a one time bug, and it can't be reproduced, the bug only affect this  application and don't affect the following openshift operations.

'rhc cartridge add mysql -a php1' failed with error 'ERROR 2013 (HY000): Lost connection to MySQL server at 'reading initial communication packet', system error: 104'.
After that: The create database roll back faile.
The mysql can't be added to this application. After restart the applicaiton, mysql still can't be added.

Version-Release number of selected component (if applicable):
puddle-2-0-2-2014-01-16 

How reproducible:
No

Steps to Reproduce:
1.
2.
3.

Actual results:
#1. failed to add mysql
hanli1@broker ~]$ rhc cartridge add mysql -a php1
Using mysql-5.1 (MySQL 5.1) for 'mysql'
Adding mysql-5.1 to application 'php1' ... 
ERROR 2013 (HY000): Lost connection to MySQL server at 'reading initial communication packet', system error: 104
Failed to create database
#2. add mysql again
[hanli1@broker ~]$ rhc cartridge add mysql -a php1
Using mysql-5.1 (MySQL 5.1) for 'mysql'
Adding mysql-5.1 to application 'php1' ... Unexpected error: Failed to create the following private endpoints due to existing process bindings: OPENSHIFT_MYSQL_DB_HOST(3306)=127.6.24.130;
#3. try to visit mysql in app.
[php1-hanli1dom.oseanli.com 52ddd9ff6fa30cc8cf00001b]\> mysql
ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)
[php1-hanli1dom.oseanli.com 52ddd9ff6fa30cc8cf00001b]\> exit
exit
#4. list process in app
[php1-hanli1dom.oseanli.com 52ddd9ff6fa30cc8cf00001b]\> ps -ef
UID        PID  PPID  C STIME TTY          TIME CMD
3121      6638     1  0 19:23 ?        00:00:00 /usr/sbin/httpd -C Include /var/lib/openshift/52ddd9ff6fa30cc8cf00001b/php//configuration/etc/conf.d/*.conf -f /var/lib/openshift/52ddd9ff6fa30cc8cf00001b/php//configuration/etc/conf/httpd_
3121      6640  6638  0 19:23 ?        00:00:00 /usr/sbin/rotatelogs /var/lib/openshift/52ddd9ff6fa30cc8cf00001b/php//logs/error_log-%Y%m%d-%H%M%S-%Z 86400
3121      6641  6638  0 19:23 ?        00:00:00 /usr/sbin/rotatelogs /var/lib/openshift/52ddd9ff6fa30cc8cf00001b/php//logs/access_log-%Y%m%d-%H%M%S-%Z 86400
3121      6642  6638  0 19:23 ?        00:00:00 /usr/sbin/httpd -C Include /var/lib/openshift/52ddd9ff6fa30cc8cf00001b/php//configuration/etc/conf.d/*.conf -f /var/lib/openshift/52ddd9ff6fa30cc8cf00001b/php//configuration/etc/conf/httpd_
3121      6865     1  0 19:24 ?        00:00:00 /bin/sh /usr/bin/mysqld_safe --defaults-file=/var/lib/openshift/52ddd9ff6fa30cc8cf00001b/mysql//conf/my.cnf
3121      7297  6865  0 19:24 ?        00:00:00 /usr/libexec/mysqld --defaults-file=/var/lib/openshift/52ddd9ff6fa30cc8cf00001b/mysql//conf/my.cnf --basedir=/usr --datadir=/var/lib/openshift/52ddd9ff6fa30cc8cf00001b/mysql//data/ --log-er
3121     20350 20339  0 19:42 ?        00:00:00 sshd: 52ddd9ff6fa30cc8cf00001b@pts/1
3121     20351 20350  0 19:42 pts/1    00:00:00 /bin/bash --init-file /usr/bin/rhcsh -i
3121     20539 20351  0 19:42 pts/1    00:00:00 ps -ef

#5. list app in mongo db.
> db.applications.find({"name" : "php1"})
{ "_id" : ObjectId("52ddd9ff6fa30cc8cf00001b"), "_type" : "Application", "analytics" : { "user_agent" : "rhc/1.17.5 (ruby 1.8.7; x86_64-linux) (2.3.2, ruby 1.8.7 (2011-06-30) [x86_64-linux])" }, "app_ssh_keys" : [   {       "_id" : ObjectId("52ddda226fa30cc8cf000036"),   "_type" : "ApplicationSshKey",  "component_id" : ObjectId("52ddd9ff6fa30cc8cf00001b"),  "content" : "AAAAB3NzaC1yc2EAAAABIwAAAQEA26ehffExrk9lKCrux76CtQ5XPjhJoq9ZvtzXlJ8uYkmOTPEwWf3UOzReg7kWLgNs3QoEFtCGCln/yJeaeuEam2G7k580oqs4+pL6oo/UJBdBCjqPKiC0psWR1YcbRQprbRJrQFYbuTglVwPTLlk4Ee4/e7Kz85xDuBN5vL4L16mNr+3NCvq4Lm+X4+QwzFrL3+m0VvwMO0T9oYeRBwW0i8Dlv8Z3MC4EQWJDTYD6ncayYjYR3y8jJXchJnxHnH8kVng1YuT4KHFooDzx8djPh+3/3ZniL053oZ/LcOSZFbCaLEtXmsA7Fqszq6bGVo8LMXIIe1XZRSg/dDo+3NkmJQ==",     "name" : "application-52ddd9ff6fa30cc8cf00001b",        "type" : "ssh-rsa" } ], "builder_id" : null, "canonical_name" : "php1", "component_configure_order" : [ ], "component_instances" : [ {  "_id" : ObjectId("52ddd9ff6fa30cc8cf00002f"),   "cartridge_vendor" : "redhat",  "version" : "5.3",      "component_properties" : {       },     "cartridge_name" : "php-5.3",   "component_name" : "php-5.3",   "group_instance_id" : ObjectId("52ddd9ff6fa30cc8cf00001d") } ], "component_start_order" : [ ], "component_stop_order" : [ ], "config" : { "auto_deploy" : true, "deployment_branch" : "master", "keep_deployments" : 1, "deployment_type" : "git" }, "created_at" : ISODate("2014-01-21T02:22:55.180Z"), "default_gear_size" : "small", "deployments" : [       {       "_id" : ObjectId("52ddda226fa30cc8cf000037"),   "activations" : [       1390271008.261575 ],    "artifact_url" : null,  "created_at" : ISODate("2014-01-21T02:23:04.897Z"),     "deployment_id" : "9667f98c",   "force_clean_build" : false,    "hot_deploy" : false,   "ref" : "master",       "sha1" : "e0180d6" } ], "domain_id" : ObjectId("52dce5cb6fa30c32b6000006"), "domain_namespace" : "hanli1dom", "domain_requires" : [ ], "downloaded_cart_map" : {  }, "group_instances" : [      {       "_id" : ObjectId("52ddd9ff6fa30cc8cf00001d"),   "gears" : [     {       "_id" : ObjectId("52ddd9ff6fa30cc8cf00001b"),   "app_dns" : true,       "host_singletons" : true,       "name" : "php1",        "quarantined" : false,  "server_identity" : "node1.oseanli.com",        "sparse_carts" : [ ],   "uid" : 3121,   "uuid" : "52ddd9ff6fa30cc8cf00001b" } ] } ], "group_overrides" : [      {       "components" : [        {       "cart" : "php-5.3",     "comp" : "php-5.3" } ],         "min_gears" : 1,        "max_gears" : 1,        "gear_size" : "small" } ], "ha" : false, "init_git_url" : null, "members" : [   {       "_type" : null,         "n" : "hanli1",         "r" : "admin",  "f" : [         [       "domain",       "admin" ] ],    "e" : null,     "_id" : ObjectId("52dce5bd6fa30c32b6000001") } ], "name" : "php1", "owner_id" : ObjectId("52dce5bd6fa30c32b6000001"), "pending_op_groups" : [ ], "scalable" : false, "secret_token" : "nFWgx185vx0qBLaOJc50UG6FWTedFpd9NvETqlLUpWhjAaTSevFlsNw89QfnHl8rL7qKQ2abFw5HtR7LaQQqjxXFuA43KTHwahNl5NPPquhgIewsZF7u4iQRzmRqqklX", "updated_at" : ISODate("2014-01-21T02:41:04.418Z") }

#6. the broker and node log files will be attached.

Expected results:
The failure can be roll back, and cartridge can be added again

Additional info:

Comment 1 Anping Li 2014-01-21 06:22:10 UTC
Created attachment 852978 [details]
Production log

Comment 2 Anping Li 2014-01-21 06:23:09 UTC
Created attachment 852979 [details]
Platform-trace log

Comment 4 Brenton Leanhardt 2014-02-04 19:04:35 UTC
Please reopen once if you see this again and know how to reproduce the problem.  Nothing in the logs looked obvious to me.  I saw a 'no space left on device' error.  Perhaps there was a problem on the Node where this gear was installed and adding it a second time picked a new Node.  Just a guess.


Note You need to log in before you can comment on or make changes to this bug.